Log Processing

For discussion of Hand-o-matic.
User avatar
laird
Posts: 53
Joined: Thu Feb 12, 2015 4:27 pm
Location: Fort Lauderdale
Contact:

Log Processing

Postby laird » Thu Feb 19, 2015 4:12 pm

DrUsual is working on processing the Hand-o-matic logs into a format that can be analyzed to look at popularity of different sizes, etc., while protecting privacy. We've been discussing this in email, but let's move the discussion to the forum so that everyone can see what's going on, contribute, ask questions, etc.

One more question (which, btw, means you can ignore my statement earlier that there seems to be just under 250 builds...)

The first complete build in the log file starts with these rows:

Wed Jan 7 03:35:58 EST 2015


Starting Full Assembly:
Email: rogelio.ortiz@gmail.com...
Ticket: 425809787-3104772398-785212574-f9350cf
URL: type=make&prostheticHand=0&Right8=55&palmSelect=5&fingerSelect=4&gauntletSelect=3&Padding=5&part=0&email=rogelio.ortiz%40gmail.com
Params: 0 -D Left1=0 -D Left2=0 -D Left3=0 -D Left4=0 -D Left5=0 -D Left6=0 -D Left7=0 -D Left8=0 -D Left9=0 -D Left10=0 -D Right1=0 -D Right2=0 -D Right3=0 -D Right4=0 -D Right5=0 -D Right6=0 -D Right7=0 -D Right8=55 -D Right9=0 -D Right10=0 -D prostheticHand=0 -D gauntletSelect=3 -D fingerSelect=4 -D palmSelect=5 -D Padding=5 -D WristBolt=0 -D KnuckleBolt=0 -D JointBolt=0 -D ThumbBolt=0
Wed Jan 7 03:35:58 EST 2015
Completed 425809787-3104772398-785212574-f9350cf (rogelio.ortiz@gmail.com...): Wed Jan 7 03:39:33 EST 2015
adding: Left.FingerDistalMedium.stl (deflated 87%)
adding: build.log (deflated 89%)

Then it gets into all the openscad lines, etc. I was thinking that this section is "triggered" by the "Starting Full Assembly:" line, so I started by flagging those lines to identify sections for inspection.

However, the next "Starting Full Assembly:" row is followed by these lines:

Starting Full Assembly:
Email: laird@popk.in
Ticket: 425809787-4112834607-675825955-f9350cf
URL: type=make&prostheticHand=0&Right8=80&palmSelect=5&fingerSelect=4&gauntletSelect=3&Padding=5&part=0&email=laird%40popk.in
Params: 0 -D Left1=0 -D Left2=0 -D Left3=0 -D Left4=0 -D Left5=0 -D Left6=0 -D Left7=0 -D Left8=0 -D Left9=0 -D Left10=0 -D Right1=0 -D Right2=0 -D Right3=0 -D Right4=0 -D Right5=0 -D Right6=0 -D Right7=0 -D Right8=80 -D Right9=0 -D Right10=0 -D prostheticHand=0 -D gauntletSelect=3 -D fingerSelect=4 -D palmSelect=5 -D Padding=5 -D WristBolt=0 -D KnuckleBolt=0 -D JointBolt=0 -D ThumbBolt=0
Wed Jan 7 08:16:57 EST 2015
Marked as already in Progress:
Email: laird@popk.in
Ticket: 425809787-4112834607-675825955-f9350cf


Not a build. So, I'm trying to figure out the best scheme for identifying that true, full build section. Any advice on which of these methods would be best, or a Door #4?



1. Look for an openscad call with an .STL output; assume that that's going to be a full build.

2. Look for that line in the first section that says "Completed <ticketnumber>" and that indicates the build is happening.

3. Scroll through to see if there's a "[----------DONE-----------]" line before the next "Starting Full Assembly" line; if so, the section between is a full build.



Thanks,

Jeff


And my response:

I'd use the Starting Assembly web service URL. It's consistent over time, and all of the selections and field entries by users are passed from the UI to the services via the URL parameters, so that will tell you all of the selected options, which is what you want. And it'll be consistent, at least since we put the web service layer in.

From your example, the parts that I think are valuable are:

Starting Full Assembly:
Email: laird@popk.in
Ticket: 425809787-4112834607-675825955-f9350cf
URL:
type=make&prostheticHand=0&Right8=80&palmSelect=5&fingerSelect=4&gauntletSelect=3&Padding=5&part=0&email=laird%40popk.in

The same info is also in the Params logged, so parse either one:

Params: 0 -D Left1=0 -D Left2=0 -D Left3=0 -D Left4=0 -D Left5=0 -D Left6=0 -D Left7=0 -D Left8=0 -D Left9=0 -D Left10=0 -D Right1=0 -D Right2=0 -D Right3=0 -D Right4=0 -D Right5=0 -D Right6=0 -D Right7=0 -D Right8=80 -D Right9=0 -D Right10=0 -D prostheticHand=0 -D gauntletSelect=3 -D fingerSelect=4 -D palmSelect=5 -D Padding=5 -D WristBolt=0 -D KnuckleBolt=0 -D JointBolt=0 -D ThumbBolt=0

And we need the time/date:

Wed Jan 7 08:16:57 EST2015

For privacy, I'd suggest perhaps taking a hash of the email address, so that you can tell how many distinct users used HOM, so we can look at reuse, and filter out our testing. :-)

Ignore part.

We can use the ticket ID to de-duplicate the data, either as a part of your log processing or as a post-processing step (SQL 'distinct' query).

If you can chew up the logs and put the result into a CSV file to load into a database, that'd be awesomely useful. For example, it'd be easy to throw it into Google Big Query for people to query, chart, etc.

The "lookup table" for option names is these two JSON structures:

Code: Select all

{
   "prostheticHand": [
      { "id": 0, "name": "Left Hand" },
      { "id": 1, "name": "Right Hand" }
   ],
   "part": [
      { "id": 0, "name": "Assembled Model" },
      { "id": -1, "name": "Exploded View" },
      { "id": 1, "name": "Gauntlet", "filename":"CyborgBeast-Gauntlet-1" },
      { "id": 2, "name": "Palm", "filename":"CyborgBeast-Palm-1" },
      { "id": 3, "name": "Finger Proximal (Near knuckle)", "filename":"CyborgBeast-FingerProximal-4" },
      { "id": 4, "name": "Finger Distal (Fingertip)", "filename":"CyborgBeast-Fingertip-4" },
      { "id": 5, "name": "Thumb Proximal (Near knuckle)", "filename":"CyborgBeast-ThumbProximal-1" },
      { "id": 6, "name": "Thumb Distal (Thumbtip)", "filename":"CyborgBeast-Thumbtip-1" }
   ],
   "gauntlet": [
      { "id": 7, "name": "Cyborg Beast Gauntlet" }
   ],
   "finger": [
      { "id": 1, "name": "Cyborg Beast, Bumps" },
      { "id": 6, "name": "Cyborg Beast, No Bumps" }
   ],
   "palm": [
      { "id": 1, "name": "Cyborg Beast" },
      { "id": 4, "name": "Cyborg Beast: No thumb" }
   ]
}

{
   "prostheticHand": [
      { "id": 0, "name": "Left Hand" },
      { "id": 1, "name": "Right Hand" }
   ],
   "part": [
      { "id": 0, "name": "Assembled Model" },
      { "id": -1, "name": "Exploded View" },
      { "id": 2, "name": "Palm", "filename":"Raptor-Palm-1" },
      { "id": 1, "name": "Gauntlet", "filename":"Raptor-Gauntlet-1" },
      { "id": 3, "name": "Finger Proximal (Near knuckle)" , "filename":"Raptor-FingerProximal-5"},
      { "id": 8, "name": "Finger Distal (Fingertip) Short", "filename":"Raptor-FingerDistalShort-1" },
      { "id": 4, "name": "Finger Distal (Fingertip) Medium", "filename":"Raptor-FingerDistalMedium-3" },
      { "id": 9, "name": "Finger Distal (Fingertip) Long", "filename":"Raptor-FingerDistalLong-1" },
      { "id": 7, "name": "Other Parts (pins, tensioner, etc.)", "filename":"Raptor-OtherParts-1" },
      { "id": 10, "name": "Hinge Caps", "filename":"Raptor-HingeCaps-1" }
   ],
   "gauntlet": [
      { "id": 3, "name": "Raptor Hand, Supports" },
      { "id": 4, "name": "Raptor Hand, No Supports" },
      { "id": 5, "name": "Raptor Hand, Flared, Supports" },
      { "id": 6, "name": "Raptor Hand, Flared, No Supports" }
   ],
   "finger": [
      { "id": 4, "name": "Raptor Hand" },
      { "id": 5, "name": "Raptor Hand, No Supports" }
   ],
   "palm": [
      { "id": 5, "name": "Raptor Hand: Supports" },
      { "id": 6, "name": "Raptor Hand: No Supports"},
      { "id": 7, "name": "Raptor Hand: Supports, No Thumb"},
      { "id": 8, "name": "Raptor Hand: No Supports, No Thumb"},
      { "id": 9, "name": "Raptor Hand: for PVC Mount (experimental)"},
      { "id": 10, "name": "Raptor Hand: Demo Hand (with cross-bar)"}
   ]
}


palmSelect = palm, fingerSelect = finger, gauntletSelect = gauntlet, prostheticHand = prostheticHand, part=part.

The only thing this won't get is the scaling percentage, but getting that out of the openscad events will be more complex. The measurements might be more useful, actually. And we can calculate percentages from the measurements, since we know the formula we use.

Then we can load the data into a database (or Google Big Query)
- Laird Popkin

rortizze
Posts: 5
Joined: Thu Feb 12, 2015 6:46 pm

Re: Log Processing

Postby rortizze » Thu Feb 19, 2015 6:34 pm

  • Only builds in ticket directory should be parsed - timestamp preceding this may be used as anyone of them for the cumulative build varies only by a factor of 2 to 5 minutes.
  • Any preview is throw-away
  • As Laird mentioned, unique entries should be extracted. TICKET ID can be used for this purpose
  • if you see "build complete" at the top of a file, might be cause a new build log was created before the completion
  • older data is structured differently, we'll have duplication between files since I don't have a core logging system for all distributions

A singular COMPLETED build is structured as follows in the log - note the key values to be in this Color:

TIMESTAMP
Starting Full Assembly:
Email: USER_EMAIL
Ticket: TICKET-ID
// Inventory file new to last revision
Inventory File: CATEGORY-FILE.json
URL: URL_PARAMETERS
Params: PARSED_OPTIONS
TIMESTAMP

Completed TICKET-ID (USER_EMAIL): TIMESTAMP
// MANY Adding entries per build
adding: PART_FILE.stl
[ Stack ]=---------------------------------------------------
TIMESTAMP
TicketID: TICKET-ID
Inventory File: CATEGORY-FILE.json

// MANY openscad commands & output per build. Note there to be many -D commands
openscad -o /WEB_DIRECTORY/ticket/TICKET-ID/BUILD_SIDE.PART_NAME.stl -D label=LABEL ["-D"=PARAMETER VALUE]* /WEB_DIRECTORY/e-NABLE/Assembly/Assembly.scad
OPENSCAD_OUTPUT

Completed: TIMESTAMP
[ -------------------------- DONE --------------------------- ]
Emailed TICKET-ID (USER_EMAIL): TIMESTAMP

Marked as COMPLETED:
Email: USER_EMAIL
Ticket: TICKET-ID
TIMESTAMP
Params: PARAMETERS


What I believe we need to extract is:

openscad -o /WEB_DIRECTORY/ticket/TICKET-ID/BUILD_SIDE.PART_NAME.stl -D label=LABEL ["-D"=PARAMETER VALUE]* /WEB_DIRECTORY/e-NABLE/Assembly/Assembly.scad

Where we group the requests by "TICKET-ID", get the bulk -D values and get the unique "BUILD_SIDE.PART_NAME.stl" for each.

NOTE: if you don't see "Inventory File", it's because the value had not been introduced. I think I probably need to add the type of build to the resulting STL file name for future ease of distinction.

Sample:

Code: Select all

Wed Jan  1 08:42:27 EST 2015
TicketID: 425809787-ABABABABA-670302817-f9350cf
Inventory File: raptor-options.json

openscad -o /var/code/prod/Service-Station/ticket/425809787-ABABABABA-670302817-f9350cf/Left.HingeCaps.stl  -D Left1=0 -D Left2=0 -D Left3=0 -D Left4=0 -D Left5=0 -D Left6=0 -D Left7=0 -D Left8=0 -D Left9=0 -D Left10=0  -D Right1=0 -D Right2=0 -D Right3=0 -D Right4=0 -D Right5=0 -D Right6=0 -D Right7=0 -D Right8=74 -D Right9=0 -D Right10=0 -D part=10  -D prostheticHand=0 -D gauntletSelect=5 -D fingerSelect=4 -D palmSelect=10 -D Padding=5 -D WristBolt=0 -D KnuckleBolt=0 -D JointBolt=0 -D ThumbBolt=0  /var/code/prod/Service-Station/e-NABLE/Assembly/Assembly.scad

openscad -o /var/code/prod/Service-Station/ticket/425809787-ABABABABA-670302817-f9350cf/Left.OtherParts.stl  -D Left1=0 -D Left2=0 -D Left3=0 -D Left4=0 -D Left5=0 -D Left6=0 -D Left7=0 -D Left8=0 -D Left9=0 -D Left10=0  -D Right1=0 -D Right2=0 -D Right3=0 -D Right4=0 -D Right5=0 -D Right6=0 -D Right7=0 -D Right8=74 -D Right9=0 -D Right10=0 -D part=7  -D prostheticHand=0 -D gauntletSelect=5 -D fingerSelect=4 -D palmSelect=10 -D Padding=5 -D WristBolt=0 -D KnuckleBolt=0 -D JointBolt=0 -D ThumbBolt=0  /var/code/prod/Service-Station/e-NABLE/Assembly/Assembly.scad


So maybe we have the following resulting columns:
  • TIMESTAMP: Wed Jan 1 08:42:27 EST 2015
  • INVENTORY_FILE: raptor-options
  • TICKET-ID: 425809787-ABABABABA-670302817-f9350cf
  • PARAMS: D Left1=0 -D Left2=0 -D Left3=0 -D Left4=0 ... (column for each)
  • SIDE: Left
  • PARTS_BUILT: HingeCaps, OtherParts

User avatar
laird
Posts: 53
Joined: Thu Feb 12, 2015 4:27 pm
Location: Fort Lauderdale
Contact:

Re: Log Processing

Postby laird » Thu Feb 19, 2015 11:40 pm

Can we get the equivalent log file from the mirror sites?
- Laird Popkin

DrUsual
Posts: 4
Joined: Sun Feb 22, 2015 8:48 pm

Re: Log Processing

Postby DrUsual » Sun Feb 22, 2015 8:59 pm

Good afternoon,

I've got the first pass of parsing done with a focus on the full assembly sections, splitting out the items we discussed. Right now this dumps into a SQL table on my local box and I've got the results for 203 full assembly requests in an Excel pivot table. Laird mentioned porting the table to something else; we can figure that out as one of the next steps.

At the moment the data looks largely like...data, as opposed to information. :) If you guys can fill in some blanks for me, I can make the results a little more reader-friendly, as well as provide some summaries along with the raw data.

For instance...we have 91 instances of the ProstheticHand value being 1, and 112 where it's set at 0. I'm not sure what the 0 and 1 signify, though.

Likewise, I have all the values for FignerSelect, PalmSelect, GauntletSelect, Padding, Right8, Left8. They're just numbers at the moment, though.

I did hide the email addresses with GUIDs; there are 137 unique email addresses from which requests originated. You can also see the number of requests from each unique email; 37 requestors have placed multiple hand requests. Most multi-requestors make two requests. I think I see you (Les) and Ro testing the Handomatic system in there.

There are some other interesting things, such as...13 request are at scale .0714, and there are 10 more each at 1.5455 and 1.6364. Given that relatively few scale choices show up more than once, these high frequencies might be useful to know.

DrUsual
Posts: 4
Joined: Sun Feb 22, 2015 8:48 pm

Re: Log Processing

Postby DrUsual » Sun Feb 22, 2015 9:02 pm

Oh, and on the subject of scale -- when I was searching a full assembly section to be able to pull the scale, I found there were two types of scaling lines. (I used the line you highlighted in the sample snippet, Les.) For some assemblies the scale was marked as a "CB" scale, and for some it was a "CD" scale.

Again, I don't know the context, but the difference in the data is significant. Each assembly seems to have one or the other, but not both, and the output line was formatted differently between them. When parsing the scale, I set up a routine to determine which one was available, then parse accordingly.

rortizze
Posts: 5
Joined: Thu Feb 12, 2015 6:46 pm

Re: Log Processing

Postby rortizze » Mon Feb 23, 2015 6:32 pm

DrUsual wrote:For instance...we have 91 instances of the ProstheticHand value being 1, and 112 where it's set at 0. I'm not sure what the 0 and 1 signify, though.

Likewise, I have all the values for FignerSelect, PalmSelect, GauntletSelect, Padding, Right8, Left8. They're just numbers at the moment, though.


We'll have to refer you to the following where, as for example, features like "prostheticHand" have persisted through the iterations of each JSON meta content descriptors:

https://github.com/e-nable/Assembler/bl ... tions.json

Code: Select all

"prostheticHand": [
      { "id": 0, "name": "Left Hand" },
      { "id": 1, "name": "Right Hand" }
]


As a reminder, the distinct JSON files were introduced since the v1.3. Prior to that, we had a single options file. Reason for adding additional is to have each type of device contain it's own set of configurations.

Thanks for completing the work, Dr. Usual! The intent is to inject them in to google doc excel sheet equivalent for graphing and further processing.

DrUsual
Posts: 4
Joined: Sun Feb 22, 2015 8:48 pm

Re: Log Processing

Postby DrUsual » Mon Feb 23, 2015 10:03 pm

Just added the friendly names, and I'll go have a look at Google docs -- I'm more familiar with Excel than Excel-like product products. :) I should be able to at least copy and paste the data in there for now, though, then look at automating a dump somewhere.

One oddity you'll see in the friendly names -- there are a number of rows where the PalmSelect, GauntletSelect, FingerSelect, and ProstheticHand values don't map to rows in the JSON table. Previous versions, perhaps? I notice that when one of those values doesn't match, none of them match. For example, there are a number of rows where FingerSelect = 1, GauntletSelect = 7, and PalmSelect = 1, but none of those values appear in the JSON file.

More shortly, let me look into Google Docs.

DrUsual
Posts: 4
Joined: Sun Feb 22, 2015 8:48 pm

Re: Log Processing

Postby DrUsual » Mon Feb 23, 2015 10:30 pm

Just dropped the data dump into a Google sheet. Here's the viewing URL:
https://docs.google.com/spreadsheets/d/ ... sp=sharing

Looking now at how to give edit permissions. I was hoping I could start typing "Laird" and it would pop you up from my Circle contacts list, but no such luck. Looks like I may have to identify you by email or get a separate link for editing.

User avatar
laird
Posts: 53
Joined: Thu Feb 12, 2015 4:27 pm
Location: Fort Lauderdale
Contact:

Re: Log Processing

Postby laird » Fri Feb 27, 2015 4:18 am

The spreadsheet looks awesome.

The values for the selectors are from that JSON above. Here it is stripped down to what's relevant:

FingerSelect

Code: Select all

   "finger": [
      { "id": 1, "name": "Cyborg Beast, Bumps" },
      { "id": 6, "name": "Cyborg Beast, No Bumps" },
      { "id": 4, "name": "Raptor Hand" },
      { "id": 5, "name": "Raptor Hand, No Supports" }
   ],


PalmSelect

Code: Select all

   "palm": [
      { "id": 1, "name": "Cyborg Beast" },
      { "id": 4, "name": "Cyborg Beast: No thumb" },
      { "id": 5, "name": "Raptor Hand: Supports" },
      { "id": 6, "name": "Raptor Hand: No Supports"},
      { "id": 7, "name": "Raptor Hand: Supports, No Thumb"},
      { "id": 8, "name": "Raptor Hand: No Supports, No Thumb"},
      { "id": 9, "name": "Raptor Hand: for PVC Mount (experimental)"},
      { "id": 10, "name": "Raptor Hand: Demo Hand (with cross-bar)"}
   ]



GauntletSelect

Code: Select all

   "gauntlet": [
      { "id": 7, "name": "Cyborg Beast Gauntlet" },
      { "id": 3, "name": "Raptor Hand, Supports" },
      { "id": 4, "name": "Raptor Hand, No Supports" },
      { "id": 5, "name": "Raptor Hand, Flared, Supports" },
      { "id": 6, "name": "Raptor Hand, Flared, No Supports" }
   ],
- Laird Popkin


Return to “Hand-o-matic”

Who is online

Users browsing this forum: No registered users and 1 guest