Difference between pages "Manual Maintenance" and "Main Page"

From DFN Wiki
(Difference between pages)
Jump to: navigation, search
(DFNKIT camera system reset)
 
m (Fireball Observatories: add packing page with contents of the box with DFNEXT system)
 
Line 1: Line 1:
== Connecting to your camera system ==
 
If you are ever required to perform any manual maintenance, first [[Logging in locally via WiFi or Ethernet|connect to your camera]].
 
  
If you are unable to connect to the camera system [[Logging in locally via WiFi or Ethernet|locally via the network (ethernet or WiFi)]], or [[Logging in remotely over the network|remotely over the VPN]], you can log in locally using a HDMI or VGA monitor and USB keyboard.
+
Welcome to the [[wikipedia:Desert_Fireball_Network|Desert Fireball Network's]] and Global Fireball Observatory public wiki. This wiki contains documentation for the the DFN's fireball observatories and data processing procedures for collaborators and team members. The main GFO website is on https://gfo.rocks, Australian DFN website can be found at https://dfn.gfo.rocks and the DFN outreach portal is http://fireballsinthesky.com.au/.
  
===Direct Connection===
+
'''Please contribute where you can''' by uploading information, designs and files that will help others with installation, maintenance and operation. By documenting your problems and solutions here, you can help other collaborators get up and running faster. '''We need your help to get cameras off shelves and under the night sky looking for space rocks!'''
  
# Before turning on system, connect a keyboard using USB and a monitor using either HDMI or VGA *insert figure of where to connect*
+
= Fireball Observatories =
# log in as <code>root</code>
+
Documentation to help you get your observatory up and running
  
== Checking removable hard drives ==
+
* [[Contents of the box with DFNEXT system|Packing]]
General notes:
+
* [[Getting Started]]
 +
* [[Camera Maintenance|Maintenance]]
 +
* [[Camera Troubleshooting|Troubleshooting]]
 +
* [[Getting Help]]
  
* Before deploying cameras the drives should be empty - delete the data recorded eg during testing in the lab, from all drives including /data0
+
= Fireball Data =
* It is a good practice to check the how the drives are full before replacing them when servicing the observatory. If it was running for several months, at least /data1 should be full. If that is not true, the observatory was not working and needs to be serviced. Also you can decide to replace only the drives that actually have some data on them and leave the empty ones in the box.
+
How to access your fireball images and information about how the DFN processes fireball ev
  
'''Now let's start - first switch the drives on and mount them:'''
+
* [[Accessing data]] captured by Fireball Observatory (still images, videos, logs)
$ python /opt/dfn-software/enable_ext-hd.py
+
* [[Finding Fireballs]] in images
In case of '''DFNEXT''' observatories, wait at least 20 seconds fro the drives to spin up. then mount the drives:
+
* [[Data Processing Pipeline]]
$ mount /data1
 
$ mount /data2
 
$ mount /data3
 
In case of '''DFNSMALL''' observatories, wait at least 40 seconds fro the system to recognize the USB enclosure and spin up the drives and then mount them:
 
$ mount -a
 
The next step is to list the drives - we are interested only in the '''data''' partitions:
 
$ df -h
 
In case of '''DFNEXT''' observatory with three 6TB removable drives installed and running for several weeks, you will get listing like this:
 
Filesystem      Size  Used Avail Use% Mounted on
 
...
 
/dev/sda3      390G  55G  316G  15% /data0
 
/dev/sdb1      5.5T  1.1T  4.2T  21% /data1    ''..... This drive is 21% full''
 
/dev/sdd1      5.5T  58M  5.2T  1% /data2    ''..... This drive is empty''
 
/dev/sdc1      5.5T  89M  5.2T  1% /data3    ''..... This drive is empty''
 
In case of '''DFSMALL''' observatory with two 8TB removable drives installed and runnin
 
$ cd /data0/DFNXXXNN/YYYY/MM/g for more than 1/2 year, now pretty much full of data, you will see:
 
Filesystem      Size  Used Avail Use% Mounted on
 
...
 
/dev/sda5      406G  59G  327G  16% /data0
 
/dev/sdc1      7.3T  6.8T  90G  99% /data1    ''..... This drive is full''
 
/dev/sdb1      7.3T  6.5T  367G  95% /data2    ''..... This drive is nearly full''
 
''Note: the /data0 partition is on the system SSD drive, this one is available all the time and contains the recent data (last 1-2 nights) and logs.''
 
  
When done with the drives check, unmount them, tell the OS to forget about the SATA devices (it's SATA hot-swap) and finally switch them off:
+
= Meteorite searching =
 +
[[Media:Meteorite_searching_advice_handbook.pdf|Meteorite_searching_advice_handbook.pdf]]
  
In case of '''DFNEXT''' observatory:
+
= Resources =
$ python /opt/dfn-software/disable_ext-hd.py
+
Collections of things you might find useful. '''Contributions to this section are especially appreciated'''
''Note: this command actually internally calls also these commands''
 
$ umount /data1 /data2 /data3
 
$ echo 1 > /sys/block/sdb/device/delete
 
$ echo 1 > /sys/block/sdc/device/delete
 
$ echo 1 > /sys/block/sdd/device/delete
 
''so there is no need to run these individually in case of nominal conditions.''
 
  
In case of '''DFSMALL''' observatory:
+
Todo: find a better name for this section
$ umount /data1 /data2
 
$ python /opt/dfn-software/disable_ext-hd.py
 
  
==Installing new HDDs==
+
* [[Mechanical Designs]] for deploying your fireball observatory (add your stand and roof mount designs here)
 +
* [[Fireball Photos|Photos of Fireballs]] (add your pretty fireball to the galleries here at reduced resolutions for everyone to see; remember this wiki is open to the public)
 +
* [[Photos of Fireball Camera Installations]] (upload photos of your finished and in-progress installations here)
  
==== 1. Make sure the hard drives are powered off ====
+
= Wiki Info =
Also consider what time it ts - there is daily task to move data from /data0 partition to the removable drives sheduled using crontab for 10:55 local time.
+
Users with accounts may edit the wiki, and anyone may view the wiki (even without logging in).
  
==== 2. Physically replace the drives ====
+
Internal links do not work in edit mode (it's a bug), but work as expected in read mode.
Remember labeling the drives taken out of the observatory - put a sticker on the drive, note observatory type, number, site name and replacement date.
 
  
==== 3. Format the new drives after replacing ====
+
File uploads are restricted to certain types of files. If you require a new filetype to be added, please email DFN camera help.
  
===== DFNEXT observatory =====
+
Click the DFN fireball in the upper left at any time to get back to this main page.
Power on the enclosure with hard drives and start the formatting script:
 
$ python /opt/dfn-software/enable_ext-hd.py
 
wait 20 seconds, then probe the observatory type and HDDs connection type
 
$ cd /root/bin
 
$ ./dfn_setup_data_hdds.sh -p
 
prints
 
  
Probe result: DFNEXT SATA /dev/sdb data2 /dev/sdc data1 /dev/sdd data3
+
[[mediawikiwiki:Help:Editing_pages|Help Editing Wiki Pages]], [[mediawikiwiki:Help:Formatting|Formatting Help]], [[mediawikiwiki:Help:VisualEditor/User_guide|The MediaWiki VisualEditor]], [[Special:OrphanedPages|Orphaned Pages]], [[Wiki To-do List|DFN Wiki To-do List]]
  Suggested command to format all drives: /root/bin/dfn_setup_data_hdds.sh /dev/sdb data1 /dev/sdc data2 /dev/sdd data3
 
  
To format all three drives, execute the suggested command
+
= Global Fireball Observatory Status Summary =
  
$ ./dfn_setup_data_hdds.sh /dev/sdb data1 /dev/sdc data2 /dev/sdd data3
+
Last update: Aug 2019
  
''Note: The formatting procedure includes SMART selftest of all the drives.''
+
{| class="wikitable"
 
+
!Region
Tell the OS to forget about the SATA devices (it's SATA hot-swap) and power them off.
+
!Institution
$ python /opt/dfn-software/disable_ext-hd.py
+
!Built and Delivered
''Note: this command actually internally calls also these commands''
+
!Deployed
''Note: this command actually internally calls also these commands''
+
!Need Servicing
$ umount /data1 /data2 /data3
+
|-
$ echo 1 > /sys/block/sdb/device/delete
+
|🇦🇺 DFN (Nullarbor, SA, WA)
$ echo 1 > /sys/block/sdc/device/delete
+
|Curtin University
$ echo 1 > /sys/block/sdd/device/delete
+
|50
''so there is no need to run these individually in case of nominal conditions.''
+
|50
 
+
|13
===== DFNSMALL observatory =====
+
|-
Power on the enclosure with hard drives and start the formatting script:
+
|🇦🇺 New South Wales
$ python /opt/dfn-software/enable_ext-hd.py 
+
|Macquarie University
wait 40 seconds
+
|0
$ cd /root/bin
+
|0
$ ./setup_usb_hdds_jmicron.sh
+
|0
When prompted, for various settings:
+
|-
 
+
|🇦🇺 Victoria
- prompt gparted --> N
+
|Monash University
 
+
|2
- prompt "Create partition /dev/sdb1, Format /dev/sdb1 as ext4" --> Y
+
|0
 
+
|0
- prompt "Create partition /dev/sdc1, Format /dev/sdc1 as ext4" --> Y
+
|-
 
+
|🇦🇺 Australian Capital Territory
- wait for quick smart self test to finish, check result for the 1st drive, particularly the following lines:
+
|Australian National University
 
+
|2
...
+
|2
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
+
|0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
+
|-
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
+
|🇦🇺 Queensland
...
+
|University of Southern Queensland
<nowiki>#</nowiki> 1  Short offline       Completed without error       00%       586         -
+
|0
...
+
|0
- press enter, check result for the 2nd drive the same way
+
|0
 
+
|-
- At the end, check that the freshly formated drives mounted and the expected capacities are listed
+
|🇬🇧 United Kingdom
 
+
|Imperial College/Uni of Glasgow
Filesystem      Size  Used Avail Use% Mounted on
+
|6
/dev/sdb1      5.5T  58M  5.2T  1% /data2    ..... this is 6TB drive
+
|6
/dev/sdc1      3.6T  68M  3.4T  1% /data1    ..... this is 4TB drive
+
|0
And finally power off the
+
|-
$ python /opt/dfn-software/disable_ext-hd.py 
+
|🇸🇦 Saudi Arabia
 
+
|National Center for Astronomy
== Transfer data from data0 to removable HDs manually ==
+
|2
$ nohup /usr/local/bin/move_data_files.sh &
+
|2
This will store info on move in nohup.out file in the present working directory. The data moving task will continue even if you disconnect from the camera.
+
|0
 
+
|-
== Check CF card ==
+
|🇲🇦 Morocco - MOFID
To check if there are images on the camera CF card:
+
|Université Hassan II de Casablanca
$ python /opt/dfn-software/enable_camera.py
+
|11
$ gphoto2 -L -R    # this will list all files on camera
+
|6
To format the CF card, see the dedicated instruction [[Formatting the CF Card|page here]].
+
|5
 
+
|-
== Capture control test ==
+
|🇨🇦 Canada, Alberta
 
+
|University of Alberta
==== 1. Run the test ====
+
|11
$ /opt/dfn-software/int_control_test.sh
+
|6
 
+
|2
This will take ~10 mins. You should start to hear shutter clicks as test photos are taken.
+
|-
 
+
|🇨🇦 Canada, Ontario
To monitor interval test as it goes (in other terminal):
+
|University of Western Ontario
$ tail -f /data0/latest/*interval.txt
+
|9
 
+
|5
==== 2. Check interval control test successfully took pictures ====
+
|0
Check there are ~10 pictures taken at the time test was run in previous images
+
|-
$ cd /data0/latest_prev
+
|🇺🇸 Arizona, USA
$ ls
+
|University of Arizona, LPL
 
+
|5
== Configure timeozne  ==
+
|1
This command starts text GUI app that allows to set the timesone.
+
|0
$ dpkg-reconfigure tzdata
+
|-
 
+
|🇺🇸 California, USA
''Note: it is not recommended to change the timezone during the night time when the exposure capture control SW is running. I is recommended to restart the OS of the camara system embedded PC after changing the timezone. Warm reboot is OK.''
+
|SETI Institute
 
+
|18
== Reset/Reboot the camera system ==
+
|14
This command initiates warm reset:
+
|3
$ reboot
+
|-
 
+
|🇺🇸 Houston, USA
=== DFNKIT camera system reset ===
+
|Lunar and Planetary Institute
In case of DFNKIT camera systems, the 'reboot' command warm reset is the only available option for reboot. Full HW (cold) reset including the microcontroller and DSLR needs power cycling, ideally power off at the time when the system is shut down or in reboot - when a beep sounds.
+
|3
 
+
|1
=== DFNSMALL camera system reset ===
+
|1
This command initiates cold reset in case of DFNSMALL camera systems (including the microcontroller, DSLR and video camera):
+
|-
$ hard_reset.sh
+
|🇺🇸 Alabama, USA
 
+
|NASA Meteoroid Environment Office
=== DFNEXT camera system reset ===
+
|0
 
+
|0
This command initiates cold reset in case of DFNEXT camera systems (including the microcontroller, DSLR and video camera):
+
|0
$ poweroff
+
|-
 
+
|🇦🇷 Argentina
''Note: there is an electronic circuit which detects that the embedded PC is shut down, cuts the power for all sub-systems and waits a few seconds before turning the power on again.''
+
| Universidad Nacional de La Plata; GSFC NASA
 
+
|1
== Check Shutter Count==
+
|1
$ cd /data0/latest_prev   
+
|0
or folder with the format:
+
|-
$ cd /data0/DFNXXXNN/YYYY/MM/YYYY-MM-DD_DFNXXXNN_1XXXXXXXXX
+
|'''Total'''
+
|
$ exiv2 -p a **image**.NEF | grep hutter
+
|'''122'''
or
+
|'''94'''
$ grep hutter *interval.txt
+
|'''24'''
 
+
|-
== Checking GPS ==
+
|GFO except DFN
First check GPS lock and position/time information acquired by the GPS and reported to the PC - run command:
+
|
$ cgps
+
|72
See the upper part of screen for position and time. In the bottom part of the screen, the NMEA messages from GPS should be scrolling.
+
|44
 
+
|11
Press [Q] to exit cgps.
+
|}
 
+
__NOTOC__
''Note: No lock means no reception, as long as there is text scrolling in the bottom of the page, the GPS communicates with the observatory PC.''
 
 
 
Second thing to check is that the capture control SW hets the leocation and time when it runs with GPS antenna connected and with good signal reception. Inspect the *interval.txt logs, either produced by [[Configuring the Software#Capture control test|capture control test]] or by regular overnight operation. In case of nominal GPS functionality, the ntp NMEA/PPS time correction should be active:
 
 
 
INFO, interval_control_lin, ntp, +SHM(0),.NMEA.,0,l,9,16,377,0.000,-11.851,3.472
 
INFO, interval_control_lin, ntp, *SHM(1),.PPS.,0,l,9,16,377,0.000,0.022,0.008
 
and coordinates should be passed from the GPS receiver (the last number '1' means GPS has lock:
 
 
 
INFO, interval_control_lin, GPS_lonlat, 135.274305, -30.857625, 156.26, 1
 
 
 
There is also a python script to query lhe leostick/arduino microcontroller for the GPS status
 
python /opt/dfn-software/leostick_get_status.py -g
 
GPGGA,081358.000,3140.0427,S,11639.9456,E,'''1''',16,0.6,195.02,M,-23.9,M,,
 
The above GPS sentence is example of receiver having lock (bold "1"), coordinates  31 deg 40.0427 min S, 116 deg 39.9456 min E, elevation 195.02 m, while the sentence below shows situation without lock (no reception).
 
python /opt/dfn-software/leostick_get_status.py -g
 
GPGGA,045843.000,3102.8703,S,11550.3033,E,'''0''',00,99.0,202.58,M,-27.9,M,,INFO, interval_control_lin
 
== Software Updates ==
 
===Automated software updates===
 
As long as the observatory is connected to the Internet, the software that controls observatory auto updates daily from a dedicated DFN server. There are two attempts to do so in the afternoon local time, ~ 40 minutes before the daily reboot and ~ 20 minutes after; the default times are 3:30 PM and 4:30 PM for the SW update and 4:10 PM for reboot.
 
===Manual software update over Internet===
 
It might be handy to execute the network SW update manually, for example in case of testing or deploying new observatory that was off-line for some time or in transport or stored as a spare. In this case log in to the observatory and execute command
 
$ dfn_down_install_sw_from_server.sh
 
===Manual software update using local copy===
 
In case of remote site without Internet connection the only way to update the observatory software is to bring a copy of the software eg on laptop do the update locally.
 
 
 
The Australian DFN team members can find the latest stable software in the internal DFN repo in operation/SW/dfnsmall/stable (DFNSAMLL type of observatory) or operation/SW/dfnext/stable (DFNEXT type). External collaborators will be provided with a copy of the software on request.
 
 
 
Assuming the servicing person has Linux environment on her/his laptop, first step is to do a dry-run to check there are no command typo errors by running:
 
$ rsync -nrv opt usr root@172.16.1.101:/
 
If there are no errors or anomalies, just list of files that would copy, you can run the real update:
 
$ rsync -rv opt usr root@172.16.1.101:/
 
''Note: The above IP address is for local wired connection (over etherenet cable), for WiFi connection use IP <code>172.16.0.101</code>''
 
 
 
== Replacing embedded PC board ==
 
 
 
=== Commell LE-37D (DFNSMALL, DFNKIT) ===
 
 
 
When replacing the PC board, the system drive (mSATA SSD card) and optionally mobile network 3G/4G modem card is moved from the old PC to the new one.
 
 
 
==== Ethernet ports ====
 
 
 
The Ethernet networking will not work out of the box, because the udev subsystem automatically pairs the network interfaces (eth0, eth1) with unique MAC addresses for each board and remembers that setting. With the system drive in new PC board, udev recognizes the new network interfaces, but names them eth2 and eth3, which does not match with the network configuration.
 
 
 
To fix that, one needs to clear that configuration: PC needs to be booted with screen (we use cheap upcycled VGA panels in our lab most of the time but HDMI should work as well) and keyboard connected to be able to run script  /root/bin/rm_udev.sh and then reboot.
 
 
 
''Note: when the system is first booted and later a monitor connected, it most likely will not work. Keyboard should be no problem including connecting it later for a blind CTRL+ALT+DEL reboot.''
 
 
 
''Note: the udev network interface persistent config does not affect the mobile network 3G/4G modem card interface, however, the user needs to make sure the operator and modem type setting is correct for the current location.''
 
 
 
==== BIOS configuration ====
 
 
 
If using brand new unconfigured PC board or if the CMOS battery on the new PC board is replaced or if in case of unexpected camera system behaviour, make sure the [[DFNSMALL_observatories_BIOS_configuration|BIOS configuration]] is correct.
 
 
 
== Useful Commands ==
 
$ df -h                # checks if hard drives are mounted - list of mounted disk devices with disk usage/free information
 
$ lsblk                # this command lists all hard disk devices in the system and where (if) they are mounted
 
$ cgps                  # gives gps coordinates in a table if sat lock and monitors communication GPS->PC. Press [Q] to exit.
 
$ ntpq -p              # check NTP time correction status
 
$ watch df              # monitors df changes, good for checking data transfers
 
$ du -hs * | grep G    # will show folders with folders >GB
 
$ crontab -l            # shows the scheduled tasks.... good for finding commands you want to manually run now
 

Revision as of 08:34, 22 June 2020

Welcome to the Desert Fireball Network's and Global Fireball Observatory public wiki. This wiki contains documentation for the the DFN's fireball observatories and data processing procedures for collaborators and team members. The main GFO website is on https://gfo.rocks, Australian DFN website can be found at https://dfn.gfo.rocks and the DFN outreach portal is http://fireballsinthesky.com.au/.

Please contribute where you can by uploading information, designs and files that will help others with installation, maintenance and operation. By documenting your problems and solutions here, you can help other collaborators get up and running faster. We need your help to get cameras off shelves and under the night sky looking for space rocks!

Fireball Observatories

Documentation to help you get your observatory up and running

Fireball Data

How to access your fireball images and information about how the DFN processes fireball ev

Meteorite searching

Meteorite_searching_advice_handbook.pdf

Resources

Collections of things you might find useful. Contributions to this section are especially appreciated

Todo: find a better name for this section

Wiki Info

Users with accounts may edit the wiki, and anyone may view the wiki (even without logging in).

Internal links do not work in edit mode (it's a bug), but work as expected in read mode.

File uploads are restricted to certain types of files. If you require a new filetype to be added, please email DFN camera help.

Click the DFN fireball in the upper left at any time to get back to this main page.

Help Editing Wiki Pages, Formatting Help, The MediaWiki VisualEditor, Orphaned Pages, DFN Wiki To-do List

Global Fireball Observatory Status Summary

Last update: Aug 2019

Region Institution Built and Delivered Deployed Need Servicing
🇦🇺 DFN (Nullarbor, SA, WA) Curtin University 50 50 13
🇦🇺 New South Wales Macquarie University 0 0 0
🇦🇺 Victoria Monash University 2 0 0
🇦🇺 Australian Capital Territory Australian National University 2 2 0
🇦🇺 Queensland University of Southern Queensland 0 0 0
🇬🇧 United Kingdom Imperial College/Uni of Glasgow 6 6 0
🇸🇦 Saudi Arabia National Center for Astronomy 2 2 0
🇲🇦 Morocco - MOFID Université Hassan II de Casablanca 11 6 5
🇨🇦 Canada, Alberta University of Alberta 11 6 2
🇨🇦 Canada, Ontario University of Western Ontario 9 5 0
🇺🇸 Arizona, USA University of Arizona, LPL 5 1 0
🇺🇸 California, USA SETI Institute 18 14 3
🇺🇸 Houston, USA Lunar and Planetary Institute 3 1 1
🇺🇸 Alabama, USA NASA Meteoroid Environment Office 0 0 0
🇦🇷 Argentina Universidad Nacional de La Plata; GSFC NASA 1 1 0
Total 122 94 24
GFO except DFN 72 44 11