FreeTrack Forum

This emerging implementation is fascinating on several levels- the cool technology, and also seeing DIY hacks leap so far ahead of commercial development- they really didn't see this coming, did they?

Many thanks to FreeTrack, for enabling so much independent development and integration.  And thanks for this forum and all of you, who are making such informative posts.

3 questions for the many here, who are far ahead of me in the learning curve:

1. Why is it that the wiimote offers higher fps (some report 100fps) and lower (1% or less) cpu load?

Is there a processor within the wii remote, providing a cleaner/simpler output to the PC?  I have cold feet about TrackIR and other webcam-based setups, because video streaming seems much too cpu-intense.  An xyz+zoom data-stream is all we're really after at the computer end.  Ideally, that's all the PC should get, without any need for processing video.

2. Has anyone experimented with wiimote clones (such as the Stix 400)?

Who cares if they have crappy accelerometers- We aren't using them anyway.

3.  What about the feasibility of a wiimote USB hack:  Is it feasible to pop it open and convert the output to USB, eliminating bluetooth?

My interest in this technology is mostly oriented to flight simulation applications like FSX and Lock On.  Pushing my so-so PC system (AMD dual-core 2700 mHz, 2MB memory, GEForce 9500GT) to the limit I'm happy with the performance, but loathe to sacrifice framerate and graphics complexity getting 6dof head tracking.

That's why I'm leaning toward the wiimote, because I want to minimize cpu usage and maximize tracking precision.  Generally from webcams to wiimotes and beyond it's a fascinating phenomenon, seeing everyday geeks, and generous developers like FreeTrack getting so far ahead of commercial product development.

I'm no programmer, but I'm handy with a soldering iron, general prototyping, and I can read schematics.  But I'd prefer not to spend considerable time and effort making adaptations, only to see a superior, cost-effective, and application-specific product hit the market before I get the setup working to my satisfaction.  Although wiimote seems the best component available now for a DIY project, it's a pity having accelerometers, bluetooth, and other circuitry included, that are superfluous to this application.  My hat is off to all of you, who have been pushing the envelope.  It displays a spirit of independent innovation that gives me hope for our future.

Nintendo authorized, but NOT recommended/utterly incompatible.  I ran into this googling for a cheap wiimote.  But it does come with fwii candy.  

Over at Procrastineering Johnny Chung Lee gets down to the specific component of the wiimote I'm most interested here

...and wires it to USB through a microcontroller (!)

That brings us back to my original question:  Why is the wii output less resource-hungry on the PC end, than webcams are?

Looking for answers, I also discovered wiibrew and Wiimote Wiki in the Procrastineering page linked above. There, we have deeper looks inside the wiimote- and maybe somewhere beyond.

But I'm still unclear on what the most efficient hardware/firmware between the CCD/CMOS camera, and software like FreeTrack is.  If I understand JCL's blog correctly, the micro camera is doing image processing, and his microcontroller is just interfacing it with USB.  Is that it?  If so, is JCL's little board the first of its kind, or is there already a small integrated camera/microcontroller out there that will outperform webcams for tracking applications???

As I already mentioned, I'm not interested in the wiimote's accelerometers, force-feedback, speaker, lights, bluetooth, etc.  I doubt I'm the only one who just wants a great tracking input that won't hog a) CPU time and b) $$.  Nintendo can keep all the other widgets for all I care.

So maybe the "wii" PixArt camera is worth a look on its own...

Maybe I'll be firing up my soldering iron after all.

A further thought on where Johnny Chung Lee left off (because he's apparently moved on to other things).

The CCS C Compiler for the PIC18F4550 includes USB-HID sample code. It's simply a matter of stuffing the data you got from the PixArt camera into the input report buffers for the USB. With this, you could actually create a USB mouse profile and make it control the cursor without any software or drivers at all. If set it up as a full speed device, it's possible to get 1ms reports providing extremely low latency updates. CCS provides relatively affordable PIC programmers as well. Explaining how to set all this up is not within the scope of this post, but it should be plenty to get you started...

It would seem plenty to get some more brainstorming started.  Assuming one might wire and program an interface between (for instance) the Wii's PixArt camera and the USB guts of a basic 4-axis joystick, could we produce a different way of bringing in head tracker values to the PC, that avoids burdening the computer with video processing...  ?

oops double post...

I'm interested to know (can anyone here explain?) what the signals from the Wii-cam/PixArt really describe.  Does that camera-chip actually express points and polygons (or relative positions of points) in terms of shape and scale without any processing necessary on down the line?

What I'm trying to get straight (for my own understanding) is how and where exactly we get from camera pixels to the YPRXYZ data expressed in the FreeTrack head model.  It seems that the information can get crunched to YPRXYZ very early in the process by pattern-recognition chips.  Can Alter's algorithm reside on a chip, and can pattern-recognition reside right on a camera chip? ??

If so, this has interesting implications for headtracking and beyond. We may be on the cusp of describing complex and dynamic aspects of reality, and meshing them into VR with very little latency and noise.  Then it's only natural that we embark on mapping and enhancing the firing of our own cerebral neurons this way...

But here and now, pondering FT's Grateful-Dead Max-HeadRoom, there's food for thought:  If I understand the webcam setups correctly, they don't do things that way.  Instead, they use a desktop or laptop's processors to crunch some hefty bandwidth down into the the pitch/roll/yaw/x-y-z signals that we're after in 6dof head-tracking, nest-ce pas?

Head trackers should talk to our computers as plainly as a joystick.  A picture may be worth a thousand words, but that's too high a price in bandwidth.

Don't Know - Jianliang
-"I DONT KNOW man HoW??
I failed~Why did the IR camera from Wiimote keeps giving me 0xFF??? "

Obviously, we're not at the point where we can just run down to Radio Shack and buy this mysterious little gadget.  I can't find any reference to "MOTSensor" at the PixArt website.  Googling "PixArt MOTsensor" yields various references like these in descriptions of the WiiMote and its contents:

IR Camera Tracker
Manufactured by PixArt Imaging
Multi-Object Tracking™ engine (MOT sensor™)

Official specifications are confidential, but…

Hardware IR blob tracking up to 4 points
Resolution: 1024x768 (true: 128x96?)
Refresh Rate; 100Hz
Dot size: 4-bits
Intensity: 8-bits (Full mode)
Bounding Box: 7-bits x-y (Full mode)
Horizontal Field of view: 45 degrees

PixArt's Multi-Object Tracking™ engine (MOT sensor™) technology can track multiple objects in an unbelievably quick and responsive way.

PixArt also provides object tracking technology for Nintendo’s new-generation gaming controller “Wii.” PixArt’s multi-object tracking engine (MOT sensor) technology can track multiple objects quickly and spontaneously.

I've found no retail source for a MOTSensor (other than desoldering one from a WiiMote).  Is that really what it comes down to?  If so, I'm sure lots of kids will be breaking their WiiMotes before the year is out.  Maybe I should offer a modest bounty to some folks at the local recycling center.

Hi Padestaka and welcome abroad !

I'll try to answer your questions but I'm not a wiimote professional so it's just what I've learnt through user's post in the forum...

1. The wiimote just sees the IR points from the sensor bar (furnished with the Wii) or from your leds system (cap or clip). It's built so it only sees these points and nothing else. Afterward, I think I've read somewhere that the wiimote send to the PC only datas about the points position meanwhile a classic webcam send a video stream. That's what makes the difference about CPU ressources cosumption : with a classic webcam, the PC has to extract data about the points position from a video stream whereas it just has to read these data from the wiimote when using a wiimote.

2. We've got bad feedback about wiimote clones. Twice it has been reported they can't track more than 2 points : it's enough for a wii but it's not for  FreeTrack... Read this post : http://forum.free-track.net/index.php?showtopic=1329

3. About USB interfacing with the Wiimote, I can't help you at all, I'm sorry...

Thanks for your researches and your enthusiasm, I hope you'll find all the answers you need ! And merry Xmas !

Rémy

Thanks and for your reply, didja.  That confirms what I'm excited about- that there is means to avoiding all that video processing on the part of gamers' hard-pressed PCs.

I'm mostly over my head here, and scouring the internet for the work of smarter people than myself- But if I could get a clear sense of what to put together, I may attempt it. Getting PixArt's MOTsensor ported to USB would be a serious challenge for me. On a good day, I could probably do the physical transplant successfully- but programming the firmware (if that's necessary) would be a huge education, and probably involves getting equipment I'd rather not buy.  Now, if I knew just a little more about how Johnny Chung Lee has interfaced the 18F4550 PIC Microcontroller with the PixArt (a wiring schematic) and if the USB output could talk to FreeTrack (???)... that would really be something.




Jianliang's frustration, JCL's cryptic glimpse, and the general dearth of information about this device are apparently no accident. Someone is obviously holding back information, and limiting public access to these point-tracking, bandwidth-slashing chips.

But it's not a very complicated pinout from the pixart sensor/converter. What an amazing little device!

JLC a dit:

...one sample from the camera containing 12 bytes, 3 for each of the 4 potential points. The format of the data will be the Extended Mode (X,Y, Y 2-msb, X 2-msb, Size 4-bits). The wait timings approximate what the Wiimote does. I've called this routine 1000 times per second without ill effect. Though, I doubt this is actually scanning the sensor and instead is just reporting the contents of an interal buffer. But, people claim 200Hz updates are possible.

Having just "discovered" these new head-tracking possibilities for myself,  I'm reminded of my experience when I got involved in Radio Controlled flying models a few years ago.  2.4 gHz technology was just emerging, and it was a significant leap in technology.  But certain companies were trying to keep the implementation proprietary.  I made a successful conversion of my then-advanced FM radio to that superior wireless technology (by using knowlege shared on a board like this)  because I just couldn't wait for companies to work out their greed issues. What resulted was very cool, and bleeding-edge for about a year, and then the corporate deals were made, and 2.5 gHz wireless for RC swept the market.

This situation is even more tantalizing to me, because I suspect that a similar change is coming to gaming head trackers.  How long corporate directors and their lawyers are going to hold back progress in implementing MOT sensors in head-tracking device applications is hard to say-  they've already been holding back for some time (Nintendo's "strategic agreement" with PixArt happened 3 years ago).

Thanks for the welcome, and Happy New Year, didja-  I look forward to reading and learning more from the many smart and resourceful people around here.

Interface links:
Arduino serial-port prototyping platform
Parallax BASIC Stamp microcontroller

More links...

Kako does it- But the nitty-gritty is in Japanese! He's got more "stuff" than JLC's USB interface- But Now We're Really Getting Somewhere!!!
Kako Video - tracking w/ PixArt Sensor, without the Wiimote!




Thank you, Google Language Tools!  Here's Kako's amazing blog translated into comprehensible English! Woooooot!

Wii Infrared Sensor Connected to Microcontroller

Makezine also points to Kako's blog, and GLT English transcription.

OK I'm off to digest all this information!!!

It seems there may be enough information for someone at FreeTrack to look into whether Kako's device output is FreeTrack compatible... didja??

I'm settled on ordering a wiimote instead of a webcam.  I'm confident it will works as well as reported here through bluetooth, so I'll have to get another specimen for experimenting with, when I get really tired of looking at the ungainly wiimote atop my monitor.

Since the wiimote is fairly complex and will see rough use in its originally-intended application, I wonder about a source for dead ones to pick the bones of.   Thousands of functional MOT sensor/processors need not go into landfills (but probably will).  If some of these castoffs could be intercepted, it might make a worthwhile cottage-industry (like marcusw's Pre-built Cap service) to extract these and turn out superior USB head-tracking units-  at least until next-generation, MOT-sensor driven trackers hit the mass-production market, at some (inevitable but still unforeseeable) point.  If the development of such a gadget evolves here, some donation support to our FreeTrack hosts would certainly be in order.

Now I'm going to try and get up to speed on microcontrollers.  I'm hoping that something similar to this will provide a prototyping platform, with the PixArt sensor mounted to it:

The CREATE USB Interface- Great information for a newbie like me at this link!

The PIC can be programmed to behave as a joystick-type device by enumerating (telling the host computer about itself) as a game pad. The addition of sensors for input and actuators for force-feedback can provide an easy way of getting data in and out of various host applications (e.g. Max/MSP/Jitter, Pd, SuperCollider, Processing, etc). Using the standard game controller input method bypasses the need to write your own driver for the USB protocol by piggy-backing on the HID (Human Input Device) standard.

Is the FreeTrack 2.2 Wiimote connectivity using USB HID?  Do wiimote FreeTrack users install it as a joystick input device?

There are prototyping boards so equipped (including with 3.3v for the PixArt) for ~$30. These ready USB prototype boards are really interesting, because any necessary signal processing can be programmed from the PC though USB with freeware utilities- no investment in any further programming hardware or software needed.

Obviously, I'm learning in reverse-engineering mode here, having dropped out of engineering school to go fly airplanes.  Many years ago.  So for my educationally-challenged brain, a DIY USB joystick tutorial at FlightSim.Com is an example of how I'm mostly learning the USB side of this.

Knowing how enumeration is done right now with the Wiimote (such as the USB ReportDescriptor coding) would be very helpful in getting off on the right track.

Hey padestaka !

That's amazing how much you've learnt and how quickly you've learnt it !

But I'm sorry I can't help you any further... I don't know a thing about programming and how FreeTrack is coded... I mean I'm just a poor webmaster (and that's a big word, I've just learnt a few things about html and php, no real webmastering education...) so I can't help you with your researsches though I admire it and hope it'll get somewhere (and I'd bet it surely will !). By the way, I don't even have a wiimote myself...

Try to contact the_target (by PM and by email through the contact form on the website) who is the author of FreeTrack and also try to contact Kestrel who greaty contributed to FreeTrack evolution from beta version to what you've got now. Unfortunately, the latter has not logged on the forum for months... That's more likely them who can help you to find the answers you need. Maybe you'll have to wait a little bit cause it's holliday time and all but I guess the_target will reply back.

Good day

Rémy

Merci Rémy,  I'll PM the_target and Kestrel, and maybe they will have a look at this.

I'm as confident as I can be (as a know-nothing newb)  that the PixArt MotSensor, paired with a suitable, and suitably-programmed USB proto board can provide a high-performance tracker in a neat little package: Next-generation head-trackers now.

The webcam-based way is still cool, and I do applaud those who have and are doing that themselves.  But I have a hunch there's an even better way to do it just around the corner.

Getting the little PixArt beauties may take a little creativity- I see that there are other people out there showing more than a passing wiimote interest.  Like the fellow with this Craigslist ad:

WANTED - Broken Wii remotes $5- $10

If you have had the misfortune of breaking your Wii remote I might be able to help take a little of the frustration away. I will pay 5 to 10 dollars for your remote depending on how much it is damaged.

You can email me or call me at...

Local newspaper ads and physical bulletin-boards might be another way to lay hands on some dead Wii remotes for dissection.  Anyway, this seems like it could be a fun, worthwhile, and educational project,and I'm eager to hear from experienced hardware and software people about it.

Thanks for the tip, didja.  I'm not just sucking up, to relate one of my first impressions of this board (from before I had even started posting): The threads where you have helped people out are everywhere- you've been doing an outstanding webmastering and customer-service job for FreeTrack.

I just want to post the image of JLC's little board once more, to recap what best represents what I imagine the final product might be; the efficient little tracker I want sitting atop my monitor-  If only we could get a schematic and parts list, or if a sharp-eyed hardware person could tell us exactly what we're looking at here:


(the future?)

Noting (on review of the link way above) that Johnny Chung Lee uses the 18F4550 PIC microcontroller "which provides built-in full-speed USB capabilites", and having found this PIC tutorial, I'm setting aside the many compelling (and free) Arduino and Duemilanove resources that are online, just until I see if I can do it as nicely as Johnny did.  I'm attracted to such a pretty little board, without the (relatively) clunky, hulking Arduino/Shield apparatus.

I'm finding myself slobbering over devices like what is imaged below (which may be $45 overkill).  I'm excited by the potential for a compact, professional-looking gadget on the very first attempt.  I'm intrigued by the prospect of an arriving age when everyday people (not just MIT PhDs, or development teams, or factory directors) can adapt microcontrollers for their own creative purposes.  I expect that learning about this kind of adaptive technology may come in very handy for all sorts of cool interfaces we haven't begun to think of yet.

The pace of progress is going to change as more people are empowered to do more things, and this head-tracking techno-limbo is symptomatic of a corporate-driven society that is messing with our perceptions of what's doable and how, messing with our self-concepts of being adaptive creatures (we're programmed instead as "consumers") for their own self-serving reasons.  But I digress...  back to these intriguing little USB adapta-boards:


SBC44UC

Modtronix a dit:

The SBC44UC is an embedded (PIC based) Single Board Computer (SBC) with a full speed USB interface. It is assembled with the PIC18F4550 PIC chip from Microchip. It is programmed with the Microchip Bootloader, meaning no programmer is required to program it! It has been designed to work with the free  Microchip USB Software. This includes software for developing...Human Interface Device (HID) class firmware...

Sweeeet!! (I think)     OK, now that I've come up for air again, it's time for me to get my head back into this PIC tutorial.  I'm still mostly in the dark about what the PixArt's pin 5 & pin 6 signals will be saying and how, but that question is going to have to wait until I gain a better understanding of Kako's description of what goes on in the PixArt's registers.  Gotta study the basics some more...

Meanwhile, all you kids out there, KEEP BANGING UP THOSE WIIMOTES! WRIST-STRAPS ARE GAY! It looks like I'm going to need mass (cheap) wii casualties arriving on my workbench, so that I can ruthlessly set about taking their wii PixArt souls (mwuaaahahahahah)   .

OK hold everything-  Reviewing my own links, this all just went a little Blair-Witchy on me...  

I had glanced at the following link before, just thinking "yep that's cool" but it didn't register with me until watching it again... what... is this not...  video ?! (OK video of video)?!?  BUT I HATE VIDEO (for this application)!

Kako device output mpg

What concerns me is not the changing geometry of Kako's non-parallel incense sticks, I wasn't planning on sticking incense in my ears and mouth for IR emitters (should work in a pinch, though)

- -  Uh, Flight Leader, gotta knock it off- you look so distant, and my ears are burning again...
 /

Anyway, what I'm actually incensed horrified about is this VIDEO!    

What's going on here? Have I been sorely mistaken about what data FreeTrack gets and uses from the entire bluetooth-ported wiimote?

It's got to be a visualization, and not video.  Unless I'm very mistaken, the whole Wii-Remote / PixArt advantage is predicated on converting the information to an orientation/position data stream (closely resembling a joystick signal) at the camera, and... well video coming out would be scary now.  It's not, right? - It's just a depiction of the 2 points, like 2 cursors... Has to be.

Calibrating joysticks, I've never confused moving dots with video, but somehow I did just that, it looks so much like the webcam output (in negative lightform) that we see in FreeTrack setup.  OK,  I'm feeling pretty stupid now, should probably delete this    but I'm just going back to getting a basic education in PIC microcontroller programming.  Maybe my eyes are getting too tired from reading too much on screen.  Learning as I go here, folks...

Its a lonely business...

COME ON, I'm sure there are lurkers who could help me along some!  I'll go crazy just talking to myself and didja here  

I can not authoritatively tell you at what point a given signal is a video stream, or point data.

What I will inject is that I'd be happy if all that came out of this is a modification that replaced the Bluetooth connection with a USB connection that:
1 - negated the need for Bluetooth, presenting itself as a USB device and that FreeTrack could use it that way.
2. negated the need for batteries in the Wiimote.

While I can understand the desire for a smaller, elegant package, the full-sized Wiimote does not bother me.

An output signal that resembled joystick axis inputs could be cool.  I would think with that situation that you would not need a Freetrack program, but rather just some joystick type software that would let you adjust the curves and assign axis as needed.

If you accomplished the Bluetooth & battery removal, you would be doing a considerable service.  

Best of luck!