Home       Goals       Download       Documentation       Support Us       Development       Job Openings       Webstore       Patrons       Contact us / Enlist      

 

Mobile pointing tablet.

Intended for people suffering from Aphasia, patients dependent upon medical ventilators, for deaf to blind standard quick communication, cerebral palsy sufferers etc.

You can help furthering The Global Ability Initiative by joining the development team

Roadmap

Planned in detail:


Iteration 1

Able to display fixed size icons and multiple chapters and play sound - Formats: PNG MP3/Wav etc. - Fixed splash screens etc.


Iteration 2

Size regulation (scalability of screen and all images) - SVG compatibility - Download of splash screen with intervals from our site


Future improvement

Iteration 3 Import of new dictionary files and images from our site (i.e. changing between different languages seamlessly)


Iteration 4

Ability to edit dictionary files from within program and upload said dictionaries to your user directory if you are a registered user of the dictionary (user involvement) Import of new images from own device, scaling of said images. Upload of said images. Upload intended for either direct sharing with other users or as suggested improvement of dictionaries.


Iteration 5

Translation interface built into application to enable user assisted translation of dictionary to own language


Iteration 6

Interfacing a Cyber glove, and other i/o devices with the application to allow deaf/mutes/others to speak to the hearing via sign language and a phone.

Possibly http://www.cemetech.net/projects/item.php?id=16


Help for the developer

Videos

Images

Sounds

FAQ

Prototypes

Sound testing

In the process of being split up into modular parts according to MVC pattern - needs further cleanup, uses external javascript pages to set values and play back sounds. setsoundvalues.js & playsound.js. Python script to create setsoundvalues.js template

Uses session storage to play sound sprite Works on both Android & FF in Ubuntu Linux] - now possible to pass arguments to setanyvalue(arg1,arg2) that then starts sprite() which plays back according to value of arg1 & arg 2 which are passed into the wordgen[] array- still small click on sound change but not choppy on Android device nor on PC….

As above except no external scripts employed

Uses session storage to play sound sprite Works on both Android & FF in Ubuntu Linux]- still small click on sound change but not choppy on Android device nor on PC….

Works on both Android & FF in Ubuntu Linux - small click on sound change but not choppy on Android device nor on PC….

Sessionstoragetesting - Works to print but cant get the floats into above….

Prototype HTML5 application - Works in Firefox OGG sound, not choppy.

Prototype HTML5 application - Maybe works on Android MP3 sound, maybe not choppy, pre-loading sound and fetching already loaded element when clicked to play.

Dual sound source experiment - pre Alpha, now with test of autogenerated play functions and audio element loading generated by even uglier python hack 20111115

Layout testing

Android / iDevice - MP3 based

Newer Android devices and tablets - OGG Based - stopped working on Firefox… AAAgh

Prototype HTML application - Does what it is supposed to button wise (i.e. hyperlinking working and sound (mp3)plays while staying on same page - however choppy), scrolling working on Android Current working specs for Android system where app runs

Click to play a sound-sprite using an image - Should produce non choppy playback

Click to play a sound-sprite using multiple images - Should produce non choppy playback of different sound on different images

* back button fixed in goals & about and one added on the contact form as well so as to enable people jump back into the application from the form if they land there by accident

* Generated HTML table structure with Python 2.6 program (ugly) - remember /'s that exist in some sentences like 0/ingenting, that needs attention or HTML code will go bananas with regards to directory names.

Prototype HTML application - Does what it is supposed to button wise (i.e. hyperlinking working), and scrolling working on Android mobile also plays back sound on Android now… Darn firefox vs. webkit sound format problems… I really need to have my own Android - and a proper system to program on the android without actually touching the phone….

But anyhow - next step I guess, is to see if I can open the playback page as a onepixel or as a non visible iFrame on click…

Prototype HTML application - Does what it is supposed to button wise (i.e. hyperlinking working), and scrolling working without a hitch on Android mobile.

Once sound problem is solved the prototype will be ready for testing in the field.

Prototype HTML application - Does what it is supposed to button wise (i.e. hyperlinking working), not on mobile browsers however

Prototype HTML application - Ugly but demonstrates idea, works in regular FF i.e. can scroll through icons while menu stays in place, but does not work in Android emulator….

Prototype HTML application - Ugly but demonstrates idea, all that needs to be done is seeding the application from the files that are explained below instead of all the hardcoding in the prototype.

Prototype HTML5 application - Ugly but demonstrates idea, all that needs once menus are floating should be seeding the application based on the files that are explained below instead of all the hardcoding in the prototype. (Now have a Python parser to automatically convert the meta files to HTML, will adapt to parse from non meta files as well in order to create a full set of dictionaries)

SUPER SIMPLE Android/Mobile APPLICATION

Goal

The application must be able to show a series of images and play back sounds when buttons are pushed.

The application begins by displaying a splash icon on the main screen and potentially plays a sound. video: splash_start.MPG (please see SPLASH CONFIGURATION FILE).

The application then displays images as ICONS on screen from pictures in lists by the name chp01.txt chp02.txt etc. with a text or partial text printed underneath if present in text file.

Program must be able to handle PNG, SVG and JPG as minimum, other image formats welcome.

The chpxx.txt text files

The chpxx.txt text files are built like this: (xx represents an arbitrary number)

<entry><text>qiáo</text><imagename>P593.png</imagename><soundname>P593.wav</soundname></entry>
<entry><text>Mù jiān xiūxí</text><imagename>P367.png</imagename><soundname>P367.wav</soundname></entry>
<entry><text>Nán hái</text><imagename>P003.png</imagename><soundname>P003.wav</soundname></entry>
<entry><text>Water</text><imagename>water.png</imagename><soundname>splash.wav</soundname></entry>
<entry><text>Salt</text><imagename>shaker.svg</imagename><soundname>salt.wav</soundname></entry>
<entry><text>Herring</text><imagename>fish.png</imagename><soundname>sayherring.wav</soundname></entry>

The application should have no size limitation as such regarding number of lines in chp.txt files per se. i.e. in the number of words or sentences pr. chapter.

If text is longer than the icon, an image of a triangle at the end of the word pointing in the text direction should be shown to indicate a sentence.

This image should be user definable and called pointingright.imageformat

imageformat after the . (dot) denotes the type of image file i.e. svg, png,, jpg etc.

<entry>… </entry> indicates an entry in a text file used by the application, just like the PDF explains.

You use these to populate the main screen of the application, population style, sizes etc. is controlled by the other files as explained in the PDF.

Notice each line in the chapter file is encapsulated.

Whatever is between the <entry> tags is one entry in the textfile.

<text>… </text> the text in here, indicates the text to show if any below the icon, or it could be used as a tooltip when hovering over an icon.

<imagename>… </imagename> indicates the specific icon to display

<soundname>… </soundname> gives you the associated soundname to an icon to play when an icon is pushed.

SOUND:

The application must be able to play the sound written at the end of each line in the chpxx.txt files as is shown in the example text above, this is to happen when a user clicks on an icon on the screen. The application must as a minimum be able to playback mp3, and should be able to play ogg and wav files as well. (see video: pushing_a_button.MPG)

SCROLLING:

One needs to be able to leaf through the icons on screen in each chapter from side to side. (See video: move_icons_and_chapter_buttons.MPG)

The numbers change as you leaf through the chapters, the numbers are also buttons so that you can push to go to that chapter direct. Imagine an infinite number of chapters lined up like this [n+1][n+1][n+1][n+1][n+1][n+1][n+1] where only 5 of the chapter icons are visible at any time and as you push

The single arrow is used to start the program to automatically slowly move in the chosen direction.

One can also use the double arrow to move through the chapters (icons), tapping the double arrow twice rapidly moves all the way up or down through the possible chapters.

Suggestions for improvement of this part of the GUI without breaking overall idea welcome.

Visual presentation of content

It should be possible to define the size and number of icons displayed on screen at any given time, this should be done in a configuration file called display.txt

The syntax of the display file should be like this.

display.txt

<GENERAL>
<XNum1>2</XNum1><YNum1>3</YNum1><ENLARGESIZE>100</ENLARGESIZE>
<XNum2>3</XNum2><YNum2>3</YNum2><REDUCESIZE>95</REDUCESIZE>
</GENERAL>
<SPECIFIED>
<IMAGENAME>imagename1.jpg<ENLARGESIZE>99</ENLARGESIZE>
<IMAGENAME>imagename2.jpg<REDUCESIZE>45</REDUCESIZE>
</SPECIFIED>

<GENERAL>indicates settings for application as such

<SPECIFIED>indicates special settings for specific images

<ENLARGESIZE>an arbitrary number in percentage indicates how much images should be enlarged, 0 meaning use image as is, no size manipulation needed.

<REDUCESIZE>as above only with reduction in mind

XNum1, YNum1 - Standard mode 1 Vertical, horizontal

The number 2 is the number of images the application should show on the x-axis and the number 3, is the number of images on the y-axis, 100 indicates to enlarge image by 100%

XNum2, YNum2 - Standard mode 2 Horizontal, i.e. when the phone is tilted. (the second line inside the <GENERAL> block.

In the second line, the number 3 indicates the number of images to show on the xaxis and the number 3 indicates the number of images on the yaxis, 95 indicates a reduction in the size of the images displayed.

Enlargement of icons and reduction should be possible during live running of the application using standard zoom movements i.e. enlarge by moving index and thumb in opposite directions, and reduce by moving index and thumb towards each other.

The main screen area should expand to around 75-80% of screen size leaving the last 20-25 % to the top and bottom panels, however, one should be able to set those parameters as well in a file called screendimensions.txt

screendimensions.txt

<DISPLAYMODE_1>
<AREASIZE_1>80</AREASIZE_1>
<AREASIZE_2>10</AREASIZE_2>
<AREASIZE_3>80</AREASIZE_3>
</DISPLAYMODE_1>
<DISPLAYMODE_2>
<AREASIZE_1>90</AREASIZE_1>
<AREASIZE_2>5</AREASIZE_2>
<AREASIZE_3>5</AREASIZE_3>
</DISPLAYMODE_2>

TOP MENU/BOTTOM MENU

The top side of the screen a fixed set of buttons allow users to make quick select sections (ie. Chapters), the arrow keys should allow to scroll through chapters as I've attempted to demonstrate in the video earlier (move_icons_and_chapter_buttons.MPG). Chapter buttons are represented with a number, it should be possible to add icons to any button in the interface via a text file in the following format called ifbuttons.txt The numbers change as you leaf through the chapter, the numbers are also buttons so that you can push to go to that chapter direct, the single arrow is used to start the program to automatically slowly move in the chosen direction and the double arrow to move faster, tapping the double arrow twice rapidly moves all the way up or down through the possible chapters. The stop button (the in the top menu square) does exactly what the name indicates, namely stops the scrolling. With regards to the format of the ifbuttons.txt file it should be like this:

ifbuttons.txt

<IFBUTTONS>

<MENU1>

<PLACEMENT>TOP</PLACEMENT>

<BUTTON><BUTTON_NAME>chp1.png</BUTTON_NAME><BUTTON_TEXT>arrow1</BUTTON_TEXT>

<CORRESPONDING_COMMAND>1</CORRESPONDING_COMMAND></BUTTON>

<BUTTON><BUTTON_NAME>chp2.png</BUTTON_NAME><BUTTON_TEXT>arrow2</BUTTON_TEXT>

<CORRESPONDING_COMMAND>2</CORRESPONDING_COMMAND></BUTTON>

<BUTTON><BUTTON_NAME>stop.png</BUTTON_NAME></BUTTON>

</MENU1>

<MENU2>

<PLACEMENT>BOTTOM</PLACEMENT>

<BUTTON><BUTTON_NAME>aboutbutton.svg</BUTTON_NAME><BUTTON_TEXT>About</BUTTON_TEXT>

<CORRESPONDING_COMMAND>3</CORRESPONDING_COMMAND></BUTTON>

<BUTTON><BUTTON_NAME>exitbutton.png</BUTTON_NAME><BUTTON_TEXT>Exit</BUTTON_TEXT>

<CORRESPONDING_COMMAND>4</CORRESPONDING_COMMAND></BUTTON>

</MENU2>

</IFBUTTONS>

NOTICE: Top and bottom interface panels need to be interchangeable i.e. draggable from top to bottom and vice versa. (See video: move_menus.MPG)

All the above mentioned interface buttons have a function executed by program (corresponding command) and interface buttons should be given a unique number and listed so that they are easily discerned.

This way it is also easier to keep track of what the interface is actually doing and it will be easier to build in new functionality to the interface.

This could be done using a simple case structure

/Case which button pressed - and what to do

/Remark: 1 = load file chp1.txt CODE TO EXECUTE

/Remark: 2 = load file chp2.txt etc. CODE TO EXECUTE

/Remark: 3 = load file about.txt CODE TO EXECUTE

/Remark: 4 = exit program CODE TO EXECUTE

/End case

In the middle of the lower menu there should be a wrench icon indicating that here you can change settings, precisely how this part of the application should be is yet undecided and not part of the scope of the development.

PROGRAM EXIT & ABOUT PROGRAM

To end the application one presses the x button, at the bottom of the screen, one is then presented with a “Do you want to quit? YES/NO”, box with a YES and a NO button. If YES is pressed the program exits showing a splash screen for a certain period of time and potentially playing a sound as indicated in the SPLASH CONFIGURATION FILE. A questionmark button opposite the x when clicked presents a text that can be found in about.txt (the about.txt is the only txt file that should be in the main program directory). One can exit from this text by pressing an X button in the bottom of the text or an X button at the top.

SPLASH CONFIGURATION FILE

Both begin splash screens and end splash screens need to be defined in a file called splash.txt, also the time that the splashscreen is shown on screen needs to be defined in the file like this. At start and stop the application needs to be able to play a sound file, this is also defined in splash.txt. The file structure looks like this

name of file: splash.txt

<SPLASHCONFIGURATION>
<BEGIN_SPLASH>
<IMAGENAME>startimage.png</IMAGENAME><IMAGEDATE>20100908</IMAGEDATE><TIME>10</TIME><SOUNDNAME>splash.wav</SOUNDNAME>
<IMAGENAME>startimage2.jpg<IMAGENAME><IMAGEDATE>20100908</<TIME>5</TIME><SOUNDNAME>bzzz.wav</SOUNDNAME>
</BEGIN_SPLASH>
<END_SPLASH>
<IMAGENAME>endimage.png</IMAGENAME><IMAGEDATE>20100908</<TIME>20</TIME><SOUNDNAME>thankyou.mp3</SOUNDNAME>
<IMAGENAME>endimage2.jpg<IMAGENAME><IMAGEDATE>20100908</<TIME>3</TIME><SOUNDNAME>tune.wav</SOUNDNAME>
</END_SPLASH>
<CHECKFORSPLASH>1</CHECKFORSPLASH><LASTCHECK>20100908</LASTCHECK>
<CHECKHERE>http://globability.org/download/</CHECKHERE>
<CHECKBEGINNUMBER>1</CHECKBEGINNUMBER>
<CHECKENDNUMBER>2</CHECKENDNUMBER>
</SPLASHCONFIGURATION>

<IMAGENAME> the name of the image to be shown

<TIME> Numerical value indicates number of seconds the splash is to stay on screen.

<SOUNDNAME> indicates the name of the soundfile to play while showing the splash

<BEGIN_SPLASH>defines the begin splash block

<END_SPLASH>defines the end splash block

<CHECKFORSPLASH> indicates the time between checks for an updated splash screen, number indicates frequency 1 = daily, 2 = weekly, 3 = monthly, 0 = never

<LASTCHECK>indicates in numerical value like this 20100908 yyyymmdd when the last check was performed

<CHECKHERE>indicates the url to check for updates http://globability.org/download/

<CHECKBEGINNUMBER>indicates a check is needed for the arbitrary image number given in the begin splash block, program must count number of <IMAGE> statements and select the correct name to check for

<CHECKENDNUMBER>indicates a check is needed for the arbitrary image number given in the begin splash block, program must count number of <IMAGE> statements and select the correct name to check for

<IMAGEDATE>indicates an arbitrary number derived from the timestamp on the image, here in the example 20100908

The directory structure for the program

The directory structure for the program should be like this. /application main directory, main program files ← /images directory for images (png/jpg) includes any icon buttons the program utilizes ← /words directory for chapter files (chp.txt) ← /sound directory for sound files (mp3, ogg, wav) ← /config directory for config files (splash.txt, ifbuttons.txt) ←

The application need to be able to run independantly of where it is placed i.e. it needs to be able to be run from an arbitrary location on a device / disk. All txt files need to be plain text files that are editable by any standard text editor.

OS specification

The application needs to be developed using the newest stable Android development kit from: http://developer.android.com/index.html

Rights

All code created must be 100% open and thoroughly documented, and you will need to sign over irrevocable rights to the code as well as the irrevocable right to distribute or to alter and otherwise use under a licence of my choice.

You will yourself naturally retain the rights to said code to use as you please as long as you do not attempt to prevent the free use of the code as indicated by the agreement you sign.

You will before beginning need to confirm the above conditions.

A proper text will be provided to you to print and sign, you'll then need to scan and upload it to me and send the document by airmail to an address to be stated.

Copyright 2010 Einar Petersen All rights reserved

Previous attempts at hiring people to work The QT app - Shelved in favor if Android for now. (Nothing new here, link included for PM's sake). http://www.vworker.com/RentACoder/misc/BidRequests/ShowBidRequest.asp?lngBidRequestId=1521218 - Project cancelled (reason) (Nothing new here, link included for PM's sake)

mobile_pointing_tablet.txt · Last modified: 2013/11/24 04:41 by einarpetersen
 
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki



The contents of this site, i.e. texts, programs and images, but not limited to these, belongs to their respective owners and are protected under international copyright treaties according to individual licensing.

Please make sure you familiarize yourself with the appropriate permissions / licenses and follow these individually before using anything you find on this site.

The Global Ability Initiative is a Non Profit venture dedicated to the development of gratis, open source software for the disabled, the sick and the needy - We aim to provide assistive software to a wide variety of people, suffering for example from Aphasia, MS, ALS, Cerebral Palsy and other disabilities as well as pushing initiatives in the field of ECG monitoring.

With reservation for changes and errors.