Planetary impact flashes detection with DeTeCt software

proposed usage method

by Marc Delcroix
December 10th, 2012



News (detailed modifications history in README file)
Changes in 2.0.5 version (Jun. 29th 2016)  :
(v2.0.5) new detect_scandir.bat for permanently scanning a directory for new files to process
support for OpenSkyImager, migration to openCV 2.4.13
bugs corrected (FITS handling of DATE-OBS keyword, usage of dark file, handling of end time missing/local time in FireCapture log)

(v2.0.4) bugs corrected (Genika mid/end time at day change, old 2011 Firecapture version: date format)
(v2.0.3) Dark handling bug corrected
(v2.0.2) Shorter static executable version not requiring MSVC 2013 redistribuable and open CV DLLs

(v2.0.1) PLx capture: support for filename longer than capture's filename (X.Dupont)
(v2.0.1) handling of MMM option in Firecapture date format (A.Coffelt)
(v2.0) More robust version (no crashes) compiled under Visual Studio with candidates estimation
(v2.0) Support for all latest Firecapture 2.4, Genika 2.7.x, Sharpcap 2.1 and PLXCapture 2.3 logs
(v2.0) Better duration and datation calculation (FITS, Firecapture, ...)


Content

News
Introduction
Download

Batch mode usage
Results
Log file with datation, usage in WinJupos
How to analyze results : images generated
Communicate your positive as well as negative results !!!

Annex
Command mode usage


Introduction

The International Outer Planets Watch team has been offering since October 2012 two softwares which help testing planetary (jovian) acquisitions a posteriori looking for impacts.
The first one, "JID" is a windows software (with a graphical interface) offering to work on acquisitions while changing detection parameters.
The second one, "DeTeCt", is usable in command mode, which permits to link analysis of several acquisitions in a row without user interaction.
It supports SER, FITS, avi, wmv, bmp, jpg, tif, ... formats. It implements two detection algorithms de détection performed simultaneously:
- the first "classic" one analyses after images alignment every zone in every frame in order to check if a suspect brightness appareance occurs. It then displays a frames' interval when an impact moght have been detected.
- the second one analyses globaly the movie and generate a simple image to easily check visually if there is an impact candidate in this acquisition.

We will see here that second "DeTeCt" software, for several reasons:
    - it allows analysis of all old acquisitions in a row without user interaction (for example I kept about 1000 Jupiter acquisitions since I do planetary imaging - that is impossible to process easily with "JID", which is interactive with a graphical user interface)
    - it implements, additionally to the classic algorithm which generate false positives which have to be studied closely after processing, my own algorithm which sound to me simpler, faster and more efficient: after images alignement, it generated one image representing maximum value for each aligned pixel  in the image, minus the average value of each aligned pixel. This allows after a visual inspection for a couple of seconds to know if a temporary flash occured during acquisition. This method also has a better sensitivity permiting to potentialy detect fainter flashes.
    - once the first version out (by Luis Calderon), I pursued its development to correct bugs, improve robustness, add features which sounded interesting and very amateur oriented, and allowing to launch more easily a real impact frequency statistical analysis project with easy participation for any interested planetary amateur astronomer
    - a bonus in this tool is that in its log file, for each acquisition it indicated the middle acquisition time in winjupos format, for easy usage by 
copy-paste in measure function for example in order to use WinJupos derotation feature

Download

I hence developed a new DeTeCt version, tested successfully on thousands of acquisitions (mine and other amateurs').
If you want to test it on your own 
acquisitions download DeTeCt version 2.0.5 here.

Please send your results (DeTeCt.log file) to delcroix.marc@free.fr.

Feel free to contact me for any usage question, any error, any modification suggestion, request for supporting a new log version of your favorite acquisition softwares if they are not already supported (after checking your DeTeCt.log - see below


Here is how to use DeTeCt :


Batch mode usage

I created a file called "detect_batch.bat" which automatically analyses all sub-directories in the directory where it is, in order to:
- find ser, avi, wmv (and fits files if you define the fit root filename in the detect_batch, see below)
- launch DeTeCt for everyone of these files with "-ADUdtcdetail -minframe 32 -ignore" options
- save log file and images in a "Impact_detection" directory
You can edit this file to easily define your own parameters.


In order to batch your processing, download the .zip file on the link above  and unzip in the directory where the acquisition sub-directories are stored (jupiter_20120804 and jupiter_20121003 in the screenshot left)

Open Windows Esplorer and double-clic on "detect_batch.bat"

NB: if you need you can edit the detect_batch.bat file in a text editor, in order to modify:
- the line "set OPT=" which defines the option parameters for DeTeCt.exe
- the line "set DARKFILE=" which defines the dark filename


A command window opens,

processing start for each found acquisition,

when processing is done (more details below):
- a window with DeTeCt.log opens, file which has to be sent to me
- an explorer window opens on Impact_detection directory, where all images to be analyzed are stored

- you see "Please press a key to continue" in the command window

Here it is, everything is done, the only thing left for you is to inspect each generated image for each movie, easy, isn't it ?


Results
Processing results are the following:

- in the "Impact_detection" directory, you will find for each acquisition, with an automated generated filename from the video filename "filename_dtc_max.jpg", which is the image consisting of the max minus mean value for each pixel. This is that image which is used to detect visually if there was an impact flash candidate, see "how to analyse the images"

- in the "Impact_detection\details" directory, you will find for each acquisition, with an automated generated filename from the video filename:
1. "filename.log", consisting of the DeTeCt run output. You will find in this file the acquisition full path and name, individual frames information if you used -v option, or text indicating if an impact was found by the classic algorithm.
2. a copy of "filename_dtc_max.jpg" (see above)
3. "filename.jpg", same image but without an histogram stretch (it is darker and less "readable")
4. "filename_mean.jpg", average of all aligned images - which is used to check the "normal" aspect of the acquisition, for example the existence of satellites or dust on the sensor which can lead to false-positive detections.

- in the same directory as DeTeCt.exe file, a DeTeCt.log file which for each processed video will add a line with date, duration, ... information from  video file information or from an existing acquisition log file from Lucam Recorder, Genika, Firecapture, PLxCapture, Avi felopaul or Genicap. Please send this DeTeCt.log file to 
delcroix.marc@free.fr for a work on impact frequenvy.

Let's see now this log file in detail:

Log file with datation, usage in WinJupos

Each time DeTeCt.exe is launched (then also each time detect_batch.bat is launched), a log file called "DeTeCt.log" saved in the same directory as DeTeCt.exe will be appended with a new line per video processed, with the following information:

DeTeCt; jovian impact detection software
PLEASE SEND THIS FILE to Marc Delcroix - delcroix.marc@free.fr - for work on impact frequency (participants will be named if work is published) - NO DETECTION MATTERS!
Rating;    Start;                      End;                        Mid;                       Duration (s);  fps (fr/s);  File;                        DeTeCt version and comment
  0       ; 2014/12/03 11:41,071083 UT; 2014/12/03 11:41,903900 UT; 2014/12/03 11:41,487491 UT;   49.9690 s;  23.000 fr/s; 031214_114104.avi; detect v2.0(Dec.1,2014) (Firecapture 2.3)
  0       ; 2014/12/03 11:41,907800 UT; 2014/12/03 11:42,741667 UT; 2014/12/03 11:42,324733 UT;   50.0320 s;  23.485 fr/s; 031214_114154.avi; detect v2.0(Dec.1,2014) (Firecapture 2.3, fps calculated)
  0       ; 2014/12/03 11:42,742967 UT; 2014/12/03 11:43,063800 UT; 2014/12/03 11:42,903384 UT;   19.2500 s;  23.000 fr/s; 031214_114244.avi; detect v2.0(Dec.1,2014) (Firecapture 2.3)


For each video, there are column per column:
- Rating:                                "0"; if -ADUdtconly option is not used :  Number of candidates detected by the classic algorithm
- Start:                                   acquisition start time, with "LT" if it is in local time, "UT" in universal time or "xx" if unknown
- End:                                    acquisition end time, with "LT" if it is in local time, "UT" in universal time or "xx" if unknown
- Mid:                                    calculated acquisition mid time useable easily with copy-paste in the fields "date" and "UT" in the WinJupos measure function (beware it must be in universal time "UT")
- Duration:                              acquisition duration
- fps:                                      frames per second
- File:                                     full path and filename of the processed video
- DeTeCt version and comment:      used DeTeCt version, and a comment on how were datation and duration information estimated:
        - with file system information ("Date from file info")
        - with  information embeded in SER file ("Date from ser file info")
        - with information in acquisition software logfile (for example "Date from Lucam Recorder log"); supported software are Lucam Recorder, Genika, Firecapture, PLxCapture, Avi felopaul, Genicap. For any other log file type support (other software, or a non supported version of supported acquisition software - unfortunately formats change with versions), please send me the acquisition software log file and I will implement it quickly.
        - with date information and possibly the fps, or if duration could only be estimated (not coming directly from a log file) ("Duration estimated")
        - if a date was estimated from another one and acquisition duration ("start date estimated" or "end date estimated")


How to analyze the results : images generated in "Impact_detection" directory

positive results
Interpreting result images by example filename_dtc_max.jpg (in "Impact_detection")
(image with maximum value minus mean value for each aligned pixel)
filename_mean.jpg (in "Impact_detection\details")
(average of aligned images)
Acquisition from Anthony Wesley June 3rd 2010
(c) Anthony Wesley
(red filter, processing on a 8s extract)

Details on planet disappear totaly in the dtc_max image, showing only the impact flash, very bright.
Acquisition from Christopher Go June 3rd 2010
(c) Christopher Go
(blue filter, processing on a 2s extract)
Acquisition from Kazuo Aoki August 20th 2010
(c) Kazuo Aoki
(processing on a 2.7s extract)

Acquisition from Mazayuki Ichimaru August 20th 2010
(c) Mazayuki Ichimaru
(processing on a 3.3s extract)

Flash is obvious on detection image dtc_max, whereas hardly visible on mean image, despite it resulting not from the whole video but from a mere 3 seconds extract centered on the flash (on a one minute movie it would be totally invisible). Acquisition was made with a small 6" télescope. This algorithm shows here it's efficiency.

Acquisition from Masayuki Tachikawa August 20th 2010
(c) Masayuki Tachikawa
(processing on a 3.2s extract)
Acquisition from George Hall September 10th 2012
(c) George Hall
(
luminance filter, processing on a 4s extract)


negative results
Acquisition from Marc Delcroix September 8th 2012,
IR>685nm filter, 50s duration
(c) Marc Delcroix

Note how details are faint, nothing pops up in dtc_max image. The limb is just slightly bright (due to turbulence or alignment being not perfect).
Acquisition from Marc Delcroix May 28th 2006
(c) Marc Delcroix

No impact detected: after checking
carefully the original video, this is a hot pixel which pops up in dtc_max image. The aspect of one single pixel brught was suspect ...
Acquisition from Marc Delcroix May 6th 2006
(c) Marc Delcroix

No
impact detected: the bright zone on the limb on the right à droite is recognized as a satellite after checking ephemeris (or on other better acquisitions directly on the mean image): the satellite moving over Jupiter during acquisition, its contrast is reinforced compared to the planet on the "dtc_max" image.


The same effect can be seen if a piece of dust (dark) is on the sensor and if the planet during acquistion moves a bit: the zone on the planet where the dust was will be very dark on the mean image, but if on a few images that zone was not hidden by the dust it will appear very contrasted - bright on the dtc_max image. A satellite shadow transiting during acquisition will provoke the same effect but will be easily identified as suchye on the "mean" image.
Acquisition from Marc Delcroix July 23rd 2006
(c) Marc Delcroix

Ah, good old times of webcams ... we used the "raw" mode at these times to improve the images quality a bit. Problem was, sometimes the webcam would do an ugly "glitch" with many bright pixels paquets. Not a problem when you use an average image, but on a  dtc_max image like showed on the right it is difficult to see anything behind this pixellized band...


false positive result
Acquisition from Marc Delcroix July 30th 2011
green filter

(c) Marc Delcroix


Not to hide you anything, I believed I got one ... a small zone a little bit brighter, rather rounded shape, it lookes as one ....
But after checking frame by frame the original ser video (in Genika's SER toolbox), it revealed that a hot pixel appeared during a hundred of frames only, while the planet was turning a bit around the same pixel on the sensor. Result is a bright pixel still on a hundred of frames (the mean image of thousands of frames does not show anything), and on the dtc_max image a nice impact flash aspect ... a flase positive one! The only one in more than one thousand analyzed videos ....

Please note that the classic algorithm, as well as JID did not detect anything as their algorithm study each individual frame, contrarly to the more global algorithm.

In any case, if you want to check if there was really a flash detected, use these 3 methods:
- inspect visually the vidéo (with Registax, Genika's SER toolbox, Autostakkert, ...)
- check carefully if and what the classic algorithm has detected (lines like  "Impact detected: frame 89 (0h 00min 03s) to 100 (0h 100min 04s). Max lum 92 at frame 90")
- check what the windows graphical software JID detects

Communicate your positive as well as negative results !!!

Please send me your DeTeCt.log files (my e-mail adress delcroix.marc@free.fr is in the header of the file) : even if you do not have detected anything on your acquisitions, this information is interesting to evaluate the impact frequency on Jupiter. I will work on this topic collecting as much information possible from as many amateur astronomers as possible, this work will be presented/realized in collaboration with professionals - all amateurs participants will be duly acknowledeged and cited.

If you are lucky enoough to detect an impact, please inform me. Send your files generated by DeTeCt, and prepare to make your acquisition file available. Prepare also for a temporary celibrity, once announced to all international amateur community, medias, web sites, the pros will contact your to get more info, this is everything I wish for you!


Good luck in your search for impact flashes
!!!

Annex

Command mode usage


In principle, DeTeCt executable must be used in a DOS command window, launching it with options. In order to know these options, just type "DeTeCt" without any parameter (see left screenshot).

The default option values detect the June 2010 impact flash on the original videos.

Quick example:
    DeTeCt -ifile ./June_2010_Impact_Anthony_Wesley.avi

        For a FITs file serie, give the first filename:

    DeTeCt -ifile ./F00001.fit

Classic algorithm result:
    Impact detected: frame 89 (0h 00min 03s) to 100 (0h 100min 04s). Max lum 92 at frame 90

Additionnal examples:
    In order to generate a data file with the luminosity evolution curve which can be used to visually inspect a potential impact signature:
     DeTeCt -ifile ./June_2010_Impact_Anthony_Wesley.avi -v > luminance.dat

    In order to generate a detection image based on the differential photometry between maximal and mean value for each pixel, using  dark file dark.tif:
     DeTeCt -ifile=June_2010_Impact_Anthony_Wesley.avi -ofile=WesleyADUdtc.png -dfile=dark.tif