Strange idea for an app

[Jackolantern]

A friend on Facebook linked to a project he and a friend worked on to create a small device that could connect to everyday items to make them digital musical instruments. It was quite impressive, but of course my software-focused mind went a completely different direction.

The app I am thinking of isn't exactly a game, but more of something to play around with. It would work best on a smart-phone, such as iPhone, Android, WP7, etc. It would work with the built-in camera. You would center the camera on any item that will fit in the frame. You would use your finger to drag a box around the item you want to use, and then click a button in the app. At that point, information about the captured image (such as color, size, shape, etc.) would semi-randomly create a musical instrument profile. Then, simple manipulations of the item while the phone's camera is pointing at the item would play musical notes, such as guitar notes, drum beats, flute notes, etc. The method of interaction could be somewhat randomized into the profile as well, so that items may require being rotated to play, while others may have to be covered to various degrees and on various sides and angles with your hands, while still others may require you to protrude single fingers from around the edges of the object, or even picking the item up and moving it away and towards the phone.

Basically what this would create is an app that would let you pick any household item, capture it with your phone, and allow you to learn to play it like an atypical musical instrument! One thing is that I think it would be better to keep the "random" musical instrument profile creation to be fairly predictable so that similar items will get very similar musical instrument profiles. Perhaps there could even be a feature to record play sessions so users could, for example, upload videos showing how to play Mary Had a Little Lamb on a purple Bic pencil or Inagaddadavida on a factory Xbox controller!

Sadly, I don't have the experience to make such an app as of right now (I have very little experience working with live video feeds, although maybe some of the smart phone SDKs have features to make it easier, but then I have also never had much positive experience programming music directly), but I think it does sound fun as just a strange, silly little app. What do you guys think?

[hallsofvallhalla]

this would be pretty kewl.

[Qjedi]

It doesn't sound that difficult if you consider that the "player" could be DESIGNING the tones to come from the system.
Let's break it into something simple, consider a desktop phone:

Main body could be designated as a "small hollow" or "medium hollow" space; non-metallic, and squared. It would be designated to create a "clack" noise when struck on the edge or corner, and a "clonk" when struck on the side. It would have similar, but higher pitched, notes related to the handpiece and combined notes when using the handset to strike, plus a set of different noises/notes for the buttons.

The player could then choose a "skin" which would be your guitar riffs, drum beats, or flute notes, or even designate different sections of the item to do different types of sound (drums from the body, guitar from the handset, flutes from the keys, distortion from moving the phone closer to the game, and stretching the notes out by tilting it, just as an example)

It would pretty much HAVE to be player driven, especially at first, but you could do two releases:
1st - Players set all notes from any item, designating how shapes will sound, and how actions will affect the notes and tones.
2nd - Use the data gathered from the first release to set a baseline of how the various shapes are expected to sound in order to come up with a "standard" that people really can use intuitively.

Now... that doesn't mean it's ACTUALLY simple to do, but it seems like a reasonable way to start out.
You could also have some baseline pictures that are used with prearranged note sets, etc, for the casual player, and allow the more involved types to set their couch up as a musical instrument all on their own.
Imagine the fun of using a picture of someone for this!
Or the real person, while you test their patience to create a musical poem of them standing there looking at you in an annoyed manner!

[Slaat3]

There is an app that allows the user to interact with the computer using items filmed by the built-in video camera. As I know, it's developer wants to encourage development of other apps using this system so he made it ope-source. If you are interested I can try to find and give you more information.

[Ark]

Nice idea!

[Slaat3]

[...] Dr. Constanza offers for free on d-touch.org all the directions and softs you need to remake the experiment at your home. The professor hopes that by offering all the materials for free, he will encourage the development of this new interface [...]

[Chris]

I was at a Windows Phone app development event last Tuesday, while this might be a little of topic, I should say the ideas windows have like the live tiles etc are all pretty good ideas, but not as flexible as I would have hoped. Also the way of writing code is very windows, like the same old annoying fact that proper syntax is starting pretty much absolutely everything with a capital letter.

Moving back on topic, the API's are all very limited to what is possible and I don't think making something like this is as easy as you make it sound Jack, I happen to have worked on a codec/exporter that turned images into 3d planes. The main target for this was to help read number plates of cars from foreign countries using cameras we had mounted on police cars. It worked nearly perfect when determining number plate positions, numbers and letters on different coloured and shaped plates. I was later put into a different project group that was meant to improve the code to help it determine different objects from one another but we stumbled across one major problem, that problem was objects look very different from different angles, the vertices were nearly incomparable from different angles. The main problem that caused this was the way the shades were calculated to put darker areas further in the background. After that came the huge database we would have needed to compare 3d objects to what we could produce. This database would have needed to be huge, so huge it's nearly hard to imagine.

We came up with two conclusions. We needed a system that could scan 3d models more accurately and if it could not match the object it was looking at correctly, we would need to tell it what it is looking at (teach it).

Just before I left we started playing around with the 3d scanners that were on the market at the time, this also required using lasers to scan objects. I spent quite a lot of time messing around and testing things some of the more experienced programmers were working on.

Using lasers to scan objects seemed to be the way to go rather than the system we had put together. I don't think phones will be able to do this for a while yet, however where there's a will there is a way.

Maybe using the same technology as head tracking with cameras would work, but from what I've seen it tends to be quite slow and inaccurate.

So good luck

PS, Here's a link I found of something like what we had made, it gives the same visual representation anyway, only this also makes it into a sphere for a cool effect (requires chrome):
http://mrdoob.com/135/Sphere
Drag an image on and what it reveal itself.

[Jackolantern]

Ahh, yes, different angles would be difficult and would likely burn down the idea of having "predictable instrument profile generation". Just having the camera back a couple of inched would be hell to make the software recognize.

Thanks for all the feedback, everyone! I don't think I could make this myself anyway. I just thought it sounded cool if it could be pulled-off

[Chris]

Here's an idea that might be possible only from a different approach. Talking to the telephone! Hey isn't that what it is made for? Yes, it has a microphone, picks up the sound and sends it, it also has abilities to receive information and convert it into sounds and output that in a speaker! Whoa!

Phones have come on far more from just using sounds, you can now play fun multi-player games and all sorts. This is great but a phone still isn't optimized to its capability!

How about making an application that listens. This already exists, but is by far not as optimized as it could be, and I think there is still a really large market within this section. Functionality exists for users holding a button and then giving the phone a command. This is of course a very important feature as we don't want the phone to be constantly calling people without our intention. But it could be handy in some ways if it did listen all the time. If we could allow it to listen for longer periods of time, for example in meetings. We could try generate information at lower rates and hope that the sounds can still be clearly parsed at decent speeds, which when thinking about it shouldn't be too difficult too do.

The cool idea behind this is, it could create a basis for a whole new way of programming, but moving forward to what could be possible today. We could have our phones active during meetings or interviews, they could then listen in on conversations and try pick up on dates, numbers, anything of interest. This could be used to ease the need of pressing a button and telling your phone a date that needs to be noted, you could even make it so complex that it would know to make reminders of this date a few days in advance without you ever needing to know. Another thing that would be possible is for it to listen into music and tell you what instruments and notes are being played, the same sort of idea as Shazam if you've ever played around with it. Another idea this could be used for is teaching people how to pronounce words when learning different languages.

The list basically goes on, and in my opinion the approach to programming as a result of a system like this could turn out to be the native way of communication between devices, not only when humans operate them.

[Jackolantern]

That would be very cool! However, you can really get bogged-down in the creation of speech-recognition software. I wonder if companies like Dragon license out their impressive technology