how is it that the Format subclasses are serializable without zero-param constructors? does Java do some kind of magic with these? Do serializable classes need to be binary-compatible with JMF in their serialized form? seems like a good idea, FMJ implements this. Serialization problems: eclipse generates inline fields in some cases (as an optimization) which messes up serialization - makes it incompatible with the classes in the reference impl jar. Incompatible classes: beware of cloning issues (like size). field == vs equals. rm FormatUtils tracing and CGUtils when done. mysteries: where does Format.copy get called? from clone? encodingCode? - order of calls in equals is relevant. static initializers intersects wierdness encoding code seems like it would have probs with conflicts. optimization - all the matching done by the Manager, directly or indirectly, seems slow. I have to use file:// prefix for files with FMJ, but JMF will find the file without it. ejmf author emails: ? Rob Gordon <rgordon@pcisys.net> Steve Talley <stephen.talley@Central> http://blogs.sun.com/roller/page/talley/ ? stephen.talley@sun.com talley@boulder.Central The more I think about JFFMPEG, the more I think that by extending BasicCodec, they possibly use these native buffers which are internal sun classes. They never create these buffers, which means it is up to the Player to create them. So this codec probably only uses native buffers when used with the sun player. So if I have my own player, can I use these codecs? Perhaps with a normal buffer. Now, if my player creates some different kind of native buffer that has the same functionality and contracts as Sun's, I could use their codec with my JMF and use native buffers. But that would require me to of course provide the same BasicCodec classes to subclass from. So there appears to be no standard way to use a native buffer (not specified by the main API). If JFFMPEG were rewritten to be independent of these classes, then it could not use native buffers, since it is the Player that must provide them. How does the system locate CaptureDevices? There is a manager to get the capture device info. I think a CaptureDevice is generally a DataSource, so you get the CaptureDeviceInfo from the CaptureDeviceManager, then get the MediaLocator from the CaptureDeviceInfo, then call Manager.createDataSource with the MediaLocator. The Format has to be get/set through a FormatControl, I suppose. ------- Generic player will need a parser (e.g. QuicktimeParser) renderers and parsers do not appear to be located through the manager. So a parser, as used by Sun, is an implementation detail. Does not appear to be a javax.media... Parser interface. thru manager: protocol (DataSource) codecs? content (Handler) It is confusing because you have the file format (mov) Then the demux Then you have the codec(s) (audio, video) Then you have the renderer - this should be generic. JFFMPEG has codecs, and demux. So we need to find a video we can play with JMF/JFFMPEG. That leaves the actual Player, Renderer, and content - this is where we need the unknown handler protocol - the URLDataSource takes care of this for now. where does the parser fit in? how is the demux located? Ok, looking at JFFMPEG, you have to register: mime types codecs demux's plug ins are managed by the pluginmanager, not the manager. ok, so we put everything together using the plugin manager. prefix.content.unknown.Handler the handler (player) will be given the datasource. from the datasource, determine content type (right, we are building a graph) from the content type, find a ... parser?... find a demux using content type (demux is parser) for each stream type coming out of the demux: using the format, find a codec that has an output format that matches a renderer may have to create a graph of codecs to do this Sun's unknown handler is: com.sun.media.content.unknown.Handler Ok, so once we use JFFMPEG, then we only need the player, and the renderer. The demux and codec are provided by JFFMPEG The player only needs to link everything up. JFFMPEG doesn't work with the stream provided by URL data source. It needs to be seekable. What is mpeg4t? FOBS/FMJ has a handler: com.omnividea.media.content.unknown.Handler Not really implemented, though. FOBS problems - 64-bit linux build - JavaDecoder/NativeDecoder assume that the size is set when getting input/output formats (JavaDecoder doesn't seem to have this problem in CVS version) - Parser requires that the DataSource be the FOBS DataSource. Should support other DataSources. FOBS DataSource does not appear to be implemented fully. - com.omnividea.media.protocol.DataSource has hard-coded: public String getContentType() { return "video.ffmpeg"; } - FOBS handler gets an NPE because videoTrack is not set before syncStart. This appears to be because it is a Processor, a subclass of Player, which has the method configure(). 5/22/06 Initial civil integration is there, a lot of rough spots. correct native civil library needs to be in the library path TODO: need a browser for capture devices TODO: support hotplugging of capture devices 5/24/06 Sun appears to have an internal mime manager. Is this what is edited in their registry app? 5/31/06 http://tritonus.org/ Sun's MP3 plugin. We could extend our sound renderer to handle any format that javasound handles. But what would the demux do? which demux would we use? So a the unknown handler would get a standard data source (file), and then look for a demux with the right content type. So we could create a demux with the content type for WAV, producing raw audio output. The trick is that javasound uses normal input streams, and jmf uses its own stream interfaces with Buffers. The demux would use an AudioInputStream. Actually, we may want the codec to do the work, so that the decoding will work even if WAV audio is embedded in another container format. There are more than a few gotchas here. 1. if a codec is going to convert between audio formats using java sound, you have the problem that java sound is stream and not buffer oriented. SO the codec has to take incoming buffers and put them into a fabricated InputStream that returns the buffer data. A single input buffer may correspond to no output buffer, and the codec is forced to return OUTPUT_NOT_FILLED. The filter graph needs to be able to handle this. Not sure what the JMF filter graph does. Especially once the input is empty, and then there is probably a relatively big output buffer awaiting. 2. Where is the right place to support AU/MP3/OGG, when using javasound SPIs? demux? codec? renderer? 3. How are all the buffer sizes to be established - with audio there is no particular buffer size when coming from a file. I guess what has been confusing me also is whether WAV or AU is a "container" format - what should the demux do with it. I'm thinking now that the demux needs to provide the raw data via the tracks, but reads enough of it to determine the exact format, and perhaps does not pass the header info to the caller of track.read. What is readFrame supposed to do if it encounters an exception, like IOException? 6/29/06: Some audio files don't playback if JMF is in the classpath. MP3 - tries to use JFFMPEG. .au There is a conflict between com.ibm.media.parser.video.MpegParser and MP3 with JMF in classpath: Filter graph: audio.mpeg com.ibm.media.parser.video.MpegParser mpeglayer3, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed, 8000.0 frame rate, FrameSize=16384 bits net.sourceforge.jffmpeg.AudioDecoder LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed com.sun.media.renderer.audio.JavaSoundRenderer WITHOUT: Filter graph: audio.mpeg net.sf.fmj.media.parser.JavaSoundParser MPEG1L3, 44100.0 Hz, Mono, 38.28125 frame rate net.sf.fmj.media.codec.JavaSoundCodec LINEAR, 44100.0 Hz, 16-bit, Mono, LittleEndian, Signed, 44100.0 frame rate, FrameSize=16 bits com.sun.media.renderer.audio.JavaSoundRenderer AU with JMF in classpath: Filter graph: audio.basic com.sun.media.parser.audio.AuParser ULAW, 8000.0 Hz, 8-bit, Mono, Signed, FrameSize=8 bits com.sun.media.renderer.audio.SunAudioRenderer WITHOUT: Filter graph: audio.basic net.sf.fmj.media.parser.JavaSoundParser ULAW, 8000.0 Hz, 8-bit, Mono, 8000.0 frame rate, FrameSize=8 bits net.sf.fmj.media.codec.JavaSoundCodec LINEAR, 8000.0 Hz, 16-bit, Mono, LittleEndian, Signed, 8000.0 frame rate, FrameSize=16 bits com.sun.media.renderer.audio.JavaSoundRenderer 10/23/06 One of the things that makes JMF/FMJ programming tricky, is assumptions in the JMF architecture about when formats are known. There is kind of a chicken-and-egg problem with media: you often need to know the formats before you can build your filter graph, but you often don't know the formats until you start reading data. That makes it hard to build the filter graph correctly. JMF's unknown handler calls start on the DataSource during the realization process. It then reads 1 frame, then calls stop. It must do this to get certain info. Then, when actually starting a clip, it calls start again during prefetch. getStreams is called after connect, but before start. FMJ seems to call start twice. we start the data source during realize and we start the demux when building the filter graph, also during realize. The RawBufferParser starts the DataSource from there. The contracts in JMF are very hard to know. Who is supposed to call start on the data source? Also, what order can things be called on a Track or Stream? For example, can I get the format before/after connect, open? CIVIL: it would be nice to be able to get the format without reading a frame. 5/30/07 Processor sort of works. Still hangs at the end running Example5_4Modified Processes at media speed, not as fast as possible. July 2 2007 How would something like subtitles be handled in JMF/FMJ? Seems like something that could be a different "track" with a different format. How about DVD navigation? How about playlists, CD indexes, DVD indexes? JMF seems to be oriented towards playback of a single multi-track media element, but has no awareness of structured media like DVDs, with chapters and such. July 10 2007 TODO: - RTP transmit/transcode wizard: format selection dlg should allow multiple formats - transmit JPEG/RTP - get at bits of native players see: http://www.onjava.com/pub/a/onjava/2005/06/01/kgpjava_part2.html?page=2 see: http://lists.apple.com/archives/QuickTime-java/2006/Mar/msg00081.html - use jdshow for lti-civil Aug 24 2007 What is wrong with FMJ: - it is awkward to have plugins that span multiple functions, for example a combination DataSource, Demultiplexer and Codec. There are many cases when wrapping native libraries where it is hard to extract these into their elements, so it would be nice not to hack it to wrap a larger entity like this. Actually, for a demux/parser it is not so tricky, since the demux just produces the decoded data. However, for a multiplexer this will be more complex, since generally the calling code tells the filter graph what the input to the muliplexer should be, for example mpeg video. - there needs to be a general clarification of vague parts of the spec. There are TODOs littered around the code that point these out. For example: - when a failure occurs in opening/starting a demux, who is responsible for the cleanup? - what happens when both the EOM and discard bits are set in a buffer. Does that discard the EOM? Directshow capture: http://msdn2.microsoft.com/en-us/library/ms867162.aspx Sep 11 2007 To get FMJ to work with JMF in the classpath (JMF first), there are a few issues: 1. the FMJ registry defaults are not there - done 2. relative file URLs not supported - done 3. safexmas.mov uses a heavyweight component that paints over the url drop down Not a problem with FMJ, but noticed that playing something like internet radio doesn't work well with JMF, because it is sending lots of caching progress events, which never seem to stop... TODO: Files to work on with FMJ ahead in classpath: streaming ogg/mp3 TODO: getting 403 from //prefs.recentUrls.add("http://upload.wikimedia.org/wikipedia/commons/d/d0/Apollo_15_liftoff_from_inside_LM.ogg"); // Apollo 15 movie from wikimedia Sep 12 2007 Running with FMJ ahead in classpath: file:///home/ken/Desktop/4harmonic.mpg No audio plays, jmf.log: ## Here's the completed flow graph: net.sf.fmj.ffmpeg_java.FFMPEGParser@44323274 connects to: net.sf.fmj.media.renderer.video.SimpleAWTRenderer@f5d8d75 format: RGB, 320x240, FrameRate=25.0, 24-bit, Masks=1:2:3, PixelStride=3, LineStride=960 net.sf.fmj.ffmpeg_java.FFMPEGParser@44323274 connects to: com.sun.media.renderer.audio.JavaSoundRenderer@57f5b4d1 format: LINEAR, 32000.0 Hz, 16-bit, Mono $$ Profile: graph building: 64 ms $$ Profile: realize, post graph building: 16 ms !! DataLine not supported: LINEAR, 32000.0 Hz, 16-bit, Mono XX Unable to handle format: LINEAR, 32000.0 Hz, 16-bit, Mono This works if we remove all JMF plugins/packages from registry. It is starting to become clear that some of the JMF classes have some serious limitations, and that ultimately, we only want to use the ones that have no FMJ counterpart. Need to divide JMF plugins into good bad and ugly... TODO: http://www.archive.org/download/bunny_goes_down_2006/bunny_goes_down_2006_64kb.mp4 plays back slowly. Something to try: rtsp://ia301238.us.archive.org:554/0/items/DIA40/DIA40_64kb.mp4 http://www.archive.org/download/DIA40/DIA40_64kb.mp4 slowly - constanly reopening http stream, to see, promote URLDataSource: logger.finer("Full content type: " + conn.getContentType()); Plays, timing is bad: http://ia300129.us.archive.org/3/items/salmat/salmat.flv TODO: safexmas.mov plays twice as fast as it should. Sep 25 2007 Using AVTransmit2: JMF's stack with FMJ: RawBufferSourceStream read never gets called JMF appears to call setInputFormat on the mux for lots of possible input formats, to determine available input formats. TODO: playback of last bit of au twice preview wizard receive wizard? RTP reception doesnt' work second time. Is something not cleaning up? loop has problems type in location, then click load, then it gives 2 errors if error. preview wizard offer alaw and speex jpeg packetizer data sink for Internet radio streaming. m3u/playlists cd audio auto-loop in fmj studio support /dev/video7 ip cameras ip server player: transcode wizard maintain aspect ratio full screen snapshot plugin viewer capture control transmission stats controls (frame grabbing controls) wrap native codecs for processing synchronization implement java codecs for easy ones (jpeg, etc). controls, graphical controls details of format in capture urls native buffers Sep 27 2007 It does appear that FMJ's unknown handler will break data into chunks before feeding it to the renderer. It will take a big 1k buffer and pass in offset 0 len 500 then offset 500 len 500, or something like that. Observed when sending 1k RTP audio packets, and seeing how it calls the java sound renderer. Sep 29 2007 When FMJ requests a URL, here are the params: INFO: HDR: 'User-Agent' = 'Java/1.5.0_07' INFO: HDR: 'Accept' = 'text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2' INFO: HDR: 'Content-type' = 'application/x-www-form-urlencoded' INFO: HDR: 'Host' = 'localhost:8090' INFO: HDR: 'Connection' = 'keep-alive' Oct 8 2007 JMF's reflection-based loading mechanism is suboptimal for applets, because it then tries to load lots of nonexistant classes from the web server, that are not in the jar. Oct 10 2007 javasound demuxes need to enforce their specific formats still need to deal with the fact that most sound demuxes need to rewrite their header at the end need a way of merging formats so that if a less specific one is set, the merged format is the result. Oct 11 2007 test FmjTranscode with JMF in classpath... Oct 21 2007 Top issues: suboptimal filter graphs unspecified formats in filter graphs (negotiation with mux) civil crash audio mux hangs playback restart freeze Oct 22 2007 review of JMF javadoc, things that FMJ doesn't really use yet: prefetching CachingControl,ExtendedCachingControl should a codec block rather than return flags that call it again? Drainable? Prefetchable DataSinkListener, DataSinkEvent events RateConfiguration,RateConfigureable Mux: "If a Multiplexer implements the Clock interface, it can be used by a Player as the master time base for synchronization. In this case, the Multiplexer should update the media time and time-base time as it processes the media." Oct 29 2007 suboptimal filter graphs fixed. Still need to deal with things like image scaling conversions, and the full set of supported mux input formats. Nov 11 2007 Need YUV/RGB conversions before implementing ffmpeg encoding Mar 10 2008 Use MemoryImageSource for more efficiency.