Keystroke Biometric: Test-Taker Setup and Data Collection
Keystroke Biometric Background: read & understand this section
Last semester's project worked to develop procedures and supporting code
to perform actual test-taker authentication experiments, see
Fall 2009 Technical Report,
and this semester's project will continue that effort.
Some programming might be necessary.
This semester we will continue to develop a test-taker authentication application
that uses the current authentication system.
Your first task is to review what was done last semester and determine whether it is sufficient to run experiments.
Continue to take new data, this semester from all members of the keystroke teams.
Fast Agile XP Deliverables
We will use the agile methodology,
particularly Extreme Programming (XP) which involves small releases and fast turnarounds in roughly two-week iterations.
Some of these deliverables might be done in parallel by different members or subsets of the team.
The following is the current list of deliverables
(ordered by the date initiated, deliverable modifications marked in red,
deliverable date marked in bold red if programming involved,
completion date and related comments marked in green,
pseudo-code marked in blue):
Keystroke Deliverable 1 Instructions
Keystroke Deliverable 2 Instructions
Keystroke Deliverable 3 Instructions
- 2/3- Data Collection
For experimental purposes we need keystroke data samples over time at two-week intervals.
Each team member is to record five keystroke samples,
in alternate weeks over a nine week interval, with data sets collected in
Week 4 (Feb 11), Week 6 (Feb 25), Week 8 (Mar 11), Week 10 (Mar 25), and Week 11 (Apr 8, after Spring Break).
Thus, each team member will record a total of 25 data samples (5 samples at each of 5 recording times).
These data are to be collected using the existing data collection method.
Obtain details about using the existing data collection method from your customer Robert Zack.
Your team (Team 4: Test-Taker Setup & Data Collection) is in charge of this operation and should:
- Do not repeat the same sample "choice" (e.g., write a letter of email to a friend) during a sample collection session.
- Use the same Login information (First Name/Last Name) for each weekly sample collection session.
There is no application validation at this time; any login will be accepted.
Care should be taken to monitor and ensure that there is login consistency.
- Each set of five samples from a subject should be spaced at roughly two-week intervals,
plus or minus no more than three days.
- Send a "take data" reminder every other week to the other two teams.
- Check that data was provided properly at the end of each data-taking week.
- 2/9- Raw Data Inventory
Conduct an inventory of the keystroke raw data files.
A user, or subject, is a person entering data.
A set of samples is a group of samples by the same user produced at roughly the same time,
usually at one sitting but certainly within a few days.
Let Setn be a set of n samples, for example Set5 is a set of 5 samples.
We are interested in counting sample sets.
A large quantity of data was collected in the 2006 time frame, called early data.
More recently, since 2008, we have been collecting data over time for longitudinal studies,
called recent data, mainly sequences of sets at two week intervals.
If 3 sets of 5 samples each were recorded at 2-week intervals,
we would call these sets Set5-Week1, Set5-Week3, Set5-Week5.
Data are taken with regard to two independent variables: keyboard type (desktop keyboard or laptop keyboard),
and input mode (copy task or free text input).
This results in four keyboard-mode quadrants: Desktop Copy, Laptop Copy, Desktop Free, and Laptop Free.
Because our current interest is authenticating online test takers, we are mainly interested in the Free-text mode.
Separately count the early data and the recent data sets.
For the Desktop Free and Laptop Free quadrants, list the users by name and specify the sets.
Modify the feature extractor program (DELIVERABLE CANCELLED, given to Team 5)
The current keystroke biometric system was developed for running experiments and
is not appropriate for deployment in an actual test-taker authentication application.
The feature extraction program removes outliers and standardizes the feature measurements into the range 0-1.
These procedures perform best on large quantities of data involving many subjects
because we want the feature data for an individual subject to be unique (within the various 0-1 feature ranges)
relative to that of other subjects.
It is not reasonable, therefore, to extract features on a small amount of data,
such as a single sample from a user needing authentication, without mixing it with data from many other subjects.
However, it would be possible, though awkward and extremely inefficient, to take a sample from an individual user,
mix it with many samples from other users (such as the large training sample data),
extract features from the large mixture of samples, separate the one-user feature vector resulting from the mixture,
and finally submit that feature vector to the classifier for authentication.
The important thing is to use standardization values from a large body of data and
to use the same values for both training and testing.
For this reason, for experimental purposes we usually run the feature extractor on all the data and
then split the data into training and testing data.
To verify one student at a time we will create an efficient authentication process
to compare a single new sample against one or more enrollment samples.
Feature extraction for the training sample will be as before on a large quantity of data over many users.
The change is that the feature extractor will output the standardization x-min and x-max values for each feature.
Then, when computing the feature values for a student's new data, these recorded x-min and x-max values
will be read in and used to perform the standardization.
In summary, the revised feature extractor will have the capability of either inputting and then using,
or computing and then outputting, a file of x-min and x-max values.
Extract features on a large body of training data and output the x-min/x-max file.
Input the x-min/x-max file when extracting features on a sample from a user to be authenticated,
and send the resulting feature file together with an enrollment feature file to the classifier for authentication.
- 3/12 Updated 4/7-
Last semester's team developed a procedure for a professor to create test questions
which would then be presented to online student test-takers inside the Java applet
in stealth mode -- that is, so they would not realize their keystrokes were being captured.
Your task is to check that procedure to determine whether it works, and if it does,
inform your customer John Stewart so he can use it with his students.
You should also comment on the procedure's elegance, ease of use, etc.,
and indicate how it can be improved.
Finally, if time permits, implement the suggested improvements.
Work closely with John Stewart to ensure successful completion of simulated student testing
of the application, but using actual students in a normal course environment.
(You must determine where the data is being collected so we can find it.)
The test, which at this point will be a batch-processed simulated test,
will involve the following.
- Testing Data: collect a total of 5 data samples (simulating both enrollment and testing)
from each of 10-30 students from one of John's courses (actual students in a course environment).
- Training Data: use an existing set of data such as the 36-subject data
(the 18+18 used for training and testing in deliverables 1-3).
- Run a weak-training (different users for training and testing) experiment
- Run feature extraction on all the data, both training and testing,
then separate the output feature vector file into two files for training and testing
- Run the BAS authentication program on the training and test files to obtain performance results.
With 5 data samples per subject, this simulates 10 (5*4/2, all the possible pairs)
authentication tests per subject to give us a reasonable number for obtaining performance statistics.
- 3/25-4/3 DELIVERABLE CANCELLED, completed by Robert Zack.
We need the following ASAP so all three keystroke teams can run the experiment described in the next deliverable.
Find 18 subjects who have entered 10 keystroke data samples (all freetext on same machine type of desktop or laptop),
preferably 5 at one time and the 5 at a later time (like two weeks later).
Create two raw-data files:
the first file contains the 5 earlier-entered data samples from each of the 18 subjects,
and the second contains the 5 later-entered data samples from each of the 18 subjects.
Now, concatenate the two files to obtain one file with the earlier-entered data coming earlier in the file.
- 3/25-4/3 DELIVERABLE CANCELLED, completed by Robert Zack.
Run the feature extraction program on the raw data file from the previous deliverable to obtain a feature file,
manually split the feature file into a training file (18 subjects, first 5 feature samples from each subject)
and a test file (18 subjects, second 5 feature samples from each subject),
and run BAS on these training and test files to obtain results
similar to the common deliverables at the beginning of the semester.
The experiment at the beginning of the semester used different subjects for training and testing
(we call that "weak" training) and
this experiment uses the same subjects (but different data samples) for training and testing
(we call this "strong" training).
[Team 5 should run this experiment with both the old and the improved (real 239 features) feature extraction program.]