| 1 | <html> |
|---|
| 2 | <head> |
|---|
| 3 | <title>Logging EPICS PVs into RRD databases for WWW display</title> |
|---|
| 4 | <!-- |
|---|
| 5 | ########### SVN repository information ################### |
|---|
| 6 | # $Date: 2011-08-21 20:15:37 +0000 (Sun, 21 Aug 2011) $ |
|---|
| 7 | # $Author: jemian $ |
|---|
| 8 | # $Revision: 624 $ |
|---|
| 9 | # $HeadURL$ |
|---|
| 10 | # $Id: documentation.html 624 2011-08-21 20:15:37Z jemian $ |
|---|
| 11 | ########### SVN repository information ################### |
|---|
| 12 | --> |
|---|
| 13 | </head> |
|---|
| 14 | <body> |
|---|
| 15 | <h1>Logging EPICS PVs into RRD databases for WWW display</h1> |
|---|
| 16 | <center><small>$Id: documentation.html 624 2011-08-21 20:15:37Z jemian $</small></center> |
|---|
| 17 | |
|---|
| 18 | <h2> Overview </h2> |
|---|
| 19 | |
|---|
| 20 | <p> |
|---|
| 21 | These files and directories provide support for logging |
|---|
| 22 | EPICS numerical PVs in RRD files and maintaining a set of |
|---|
| 23 | plots for WWW browsing |
|---|
| 24 | </p> |
|---|
| 25 | |
|---|
| 26 | <p> |
|---|
| 27 | This tool records the value of an EPICS (numerical) process variable |
|---|
| 28 | and peridically records the value into an <tt>rrd</tt> database. |
|---|
| 29 | The database has been configured to present plots for the last hour, |
|---|
| 30 | last daym last week, last month, last year, and last decade. |
|---|
| 31 | A Python script monitors the PVs and updates the databases. |
|---|
| 32 | Each PV has its own database which simplifies the addition |
|---|
| 33 | or removal of PVs from interest. |
|---|
| 34 | A PHP page has been built to provide access to the suite of |
|---|
| 35 | databases from WWW clients. The <tt>index.php</tt> page |
|---|
| 36 | builds a list of available data and presents that to the WWW client |
|---|
| 37 | as a table and a representative plot. A link from each PV will |
|---|
| 38 | bring up a page with plots of the six time spans described |
|---|
| 39 | for that PV. These pages are generated by CGI scripts, |
|---|
| 40 | with information pulled immediately from what is available |
|---|
| 41 | in the database. |
|---|
| 42 | </p> |
|---|
| 43 | |
|---|
| 44 | <h2>Adding EPICS Process Variables</h2> |
|---|
| 45 | |
|---|
| 46 | <p>The EPICS Process Variables to be logged are |
|---|
| 47 | entered in the <tt>pvlist.xml</tt> file. Follow the |
|---|
| 48 | pattern for the other entries in that file. |
|---|
| 49 | Here is an example: |
|---|
| 50 | <pre> |
|---|
| 51 | <EPICS_PV pv="S:SRcurrentAI" units="mA" min="-0.1" max="310" scale="lin"> |
|---|
| 52 | APS storage ring current, mA |
|---|
| 53 | </EPICS_PV> |
|---|
| 54 | <EPICS_PV pv="33bmPLC:alarm:cg1:vac" units="Torr" min="1e-5" max="2000" scale="log"> |
|---|
| 55 | 33BM DCM convectron gauge vacuum |
|---|
| 56 | </EPICS_PV> |
|---|
| 57 | </pre> |
|---|
| 58 | </p> |
|---|
| 59 | |
|---|
| 60 | <h2>Updating EPICS PVs into RRD databases</h2> |
|---|
| 61 | |
|---|
| 62 | <p> |
|---|
| 63 | RRD needs some more parts (database and graphing scripts) |
|---|
| 64 | to be built before plots will be available on the WWW. |
|---|
| 65 | Documentation on how those parts are |
|---|
| 66 | created has not yet been written. Until then, |
|---|
| 67 | examine the files: <tt>rrdCreate.py</tt> and <tt>cgiCreate.py</tt> |
|---|
| 68 | for hints. |
|---|
| 69 | </p> |
|---|
| 70 | |
|---|
| 71 | <h2>Generating plots of logged data</h2> |
|---|
| 72 | |
|---|
| 73 | <p> |
|---|
| 74 | The APS XOR WWW server is configured for RRDTOOL support |
|---|
| 75 | to generate the plots on demand from their associated RRD databases. |
|---|
| 76 | The plots will only be generated on demand from a |
|---|
| 77 | web browser request, and then only |
|---|
| 78 | as needed by the time-granularity of the plot. |
|---|
| 79 | Thus, the plots will not be re-generated unless necessary. |
|---|
| 80 | </p> |
|---|
| 81 | |
|---|
| 82 | <p> |
|---|
| 83 | Until the XOR WWW server had rrdcgi capability, the plots and HTML pages |
|---|
| 84 | were generated by calling rrdcgi directly from a command-line job. |
|---|
| 85 | The Python program <tt>htmlCreate.py</tt> generated |
|---|
| 86 | all HTML pages and related plots from the set of CGI scripts. |
|---|
| 87 | It was run from a UNIX cron job. This is no longer used. |
|---|
| 88 | </p> |
|---|
| 89 | |
|---|
| 90 | <p> |
|---|
| 91 | RRDtool support was added to the WWW server in July 2010. |
|---|
| 92 | The WWW server needs to identify the CGI files by suffix. |
|---|
| 93 | Appending .cgi to all the CGI files will allow the files |
|---|
| 94 | to be properly identified and rendered. (tested and works) |
|---|
| 95 | </p> |
|---|
| 96 | |
|---|
| 97 | <p> |
|---|
| 98 | The plots directory needs to be writable by the WWW server user (nobody) and have the group |
|---|
| 99 | sticky bit set |
|---|
| 100 | </p> |
|---|
| 101 | |
|---|
| 102 | <h2>Configuration of the infrastructure</h2> |
|---|
| 103 | |
|---|
| 104 | <em>under construction</em> |
|---|
| 105 | |
|---|
| 106 | <p> |
|---|
| 107 | It is necessary to understand UNIX command shell scripts, |
|---|
| 108 | EPICS PV communications, php, |
|---|
| 109 | Python, rrdtool, and WWW servers to configure the |
|---|
| 110 | infrastructure of this support. |
|---|
| 111 | </p> |
|---|
| 112 | |
|---|
| 113 | <h2> crontab configuration </h2> |
|---|
| 114 | |
|---|
| 115 | <pre> |
|---|
| 116 | # field allowed values |
|---|
| 117 | # ----- -------------- |
|---|
| 118 | # minute 0-59 |
|---|
| 119 | # hour 0-23 |
|---|
| 120 | # day of month 1-31 |
|---|
| 121 | # month 1-12 (or names, see below) |
|---|
| 122 | # day of week 0-7 (0 or 7 is Sun, or use names) |
|---|
| 123 | |
|---|
| 124 | # make sure that EPICS --> RRD monitor tool is running, check every 5 minutes |
|---|
| 125 | # solaris cron does not support */5 notation from linux cron |
|---|
| 126 | 10,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/joule/WEB33/www/rrd/checkup-epics2rrd.csh 2>&1 /dev/null |
|---|
| 127 | </pre> |
|---|
| 128 | |
|---|
| 129 | <h2> assets </h2> |
|---|
| 130 | <pre> |
|---|
| 131 | cgi/ (ignored by SVN) |
|---|
| 132 | data/ (ignored by SVN) |
|---|
| 133 | html/ (no longer needed) |
|---|
| 134 | other/ |
|---|
| 135 | plots/ (ignored by SVN) |
|---|
| 136 | checkup-epics2rrd.csh |
|---|
| 137 | cgiCreate.py |
|---|
| 138 | epics2rrd.csh |
|---|
| 139 | epics2rrd.log |
|---|
| 140 | epics2rrd.pid |
|---|
| 141 | epics2rrd.py |
|---|
| 142 | htmlCreate.py (no longer needed) |
|---|
| 143 | index.php |
|---|
| 144 | Makefile |
|---|
| 145 | makeTagHTML.py |
|---|
| 146 | documentation.html |
|---|
| 147 | rrdCreate.py |
|---|
| 148 | xmlSupport.py |
|---|
| 149 | properties.xml |
|---|
| 150 | pvlist.xml |
|---|
| 151 | |
|---|
| 152 | leo% svn pg svn:ignore . |
|---|
| 153 | xref.txt |
|---|
| 154 | plots |
|---|
| 155 | cgi |
|---|
| 156 | html |
|---|
| 157 | data |
|---|
| 158 | *.log |
|---|
| 159 | *.pyc |
|---|
| 160 | |
|---|
| 161 | </pre> |
|---|
| 162 | |
|---|
| 163 | <h2 id="Status">Status</h2> |
|---|
| 164 | <p> |
|---|
| 165 | Production version is in operation for many APS operations cycles. |
|---|
| 166 | See <a class="ext-link" href="http://sector33.xor.aps.anl.gov/rrd" |
|---|
| 167 | ><span class="icon">http://sector33.xor.aps.anl.gov/rrd</span></a> |
|---|
| 168 | </p> |
|---|
| 169 | |
|---|
| 170 | <h2 id="RRDdatabasechanges">RRD database changes</h2> |
|---|
| 171 | <ul> |
|---|
| 172 | <li> |
|---|
| 173 | Each EPICS PV goes into its own RRD database. |
|---|
| 174 | This simplifies the addition of PVs into the logging system. |
|---|
| 175 | </li> |
|---|
| 176 | <li> |
|---|
| 177 | 10x finer time granularity now than before. |
|---|
| 178 | Data are being averaged into minimum 30-second bins. |
|---|
| 179 | (Data can always be updated to system at 1 Hz, if available.) |
|---|
| 180 | </li> |
|---|
| 181 | </ul> |
|---|
| 182 | |
|---|
| 183 | <h2 id="DefaultWWWpage">Default WWW page</h2> |
|---|
| 184 | <p> |
|---|
| 185 | The default WWW page is now PHP. This allows for greater flexibility |
|---|
| 186 | in designing new interfaces. The default interface shows a gallery of |
|---|
| 187 | available PNG plots with data logged from the last week. Each plot is |
|---|
| 188 | linked to a full description of logged data (6 plots for last: hour, |
|---|
| 189 | day, week, month, year, and decade). |
|---|
| 190 | </p> |
|---|
| 191 | |
|---|
| 192 | <h2 id="Codebase">Codebase</h2> |
|---|
| 193 | <p> |
|---|
| 194 | Python has been used to implement the support code, calling rrdtool |
|---|
| 195 | through the rrdtool Python support module. The main WWW page is rendered in PHP. |
|---|
| 196 | </p> |
|---|
| 197 | |
|---|
| 198 | <h3 id="Configurationdetails">Configuration details</h3> |
|---|
| 199 | <p> |
|---|
| 200 | Two XML files are used for configuration: |
|---|
| 201 | <tt>pvlist.xml</tt> describes each of the EPICS Process Variables to be logged, |
|---|
| 202 | and <tt>properties.xml</tt> describes various other configuration and implementation |
|---|
| 203 | details for the support code. |
|---|
| 204 | </p> |
|---|
| 205 | |
|---|
| 206 | <h3 id="RRDdatabasecreation">RRD database creation</h3> |
|---|
| 207 | <p> |
|---|
| 208 | Each PV has its own RRD database, created by <tt>rrdCreate.py</tt>. |
|---|
| 209 | </p> |
|---|
| 210 | |
|---|
| 211 | <h3 id="CGIscriptcreation">CGI script creation</h3> |
|---|
| 212 | <p> |
|---|
| 213 | <tt>cgiCreate.py</tt> creates CGI scripts for each PV (and corresponding RRD database). |
|---|
| 214 | </p> |
|---|
| 215 | <h3 id="RRDdatabaseupdatesfromEPICS">RRD database updates from EPICS</h3> |
|---|
| 216 | <p> |
|---|
| 217 | The task that cmonitors EPICS PVs and enters them into RRD databases is |
|---|
| 218 | called <tt>epics2rrd.py</tt>. A <em>starter script</em> (<tt>epics2rrd.csh</tt>) |
|---|
| 219 | is used to start/stop/restart the Python script, |
|---|
| 220 | record the PID in a file (<tt>epics2rrd.pid</tt>), |
|---|
| 221 | and pipe the output to a log file (<tt>epics2rrd.log</tt>). |
|---|
| 222 | </p> |
|---|
| 223 | |
|---|
| 224 | <h3 id="CGIscriptoperation">CGI script operation</h3> |
|---|
| 225 | <p> |
|---|
| 226 | A web page request will cause <tt>rrdcgi</tt> to generate |
|---|
| 227 | the www page content (text and plots) on demand. It may be necessary to force the |
|---|
| 228 | page to refresh the local browser cache in order for some of the |
|---|
| 229 | longer-term plots to be rebuilt. Usually, <shift>-"Refresh" or similar will get this done. |
|---|
| 230 | </p> |
|---|
| 231 | |
|---|
| 232 | <h2 id="Dependencies">Dependencies</h2> |
|---|
| 233 | <ul> |
|---|
| 234 | <li> |
|---|
| 235 | Python |
|---|
| 236 | <ul> |
|---|
| 237 | <li>librrdtool</li> |
|---|
| 238 | <li>CaChannel</li> |
|---|
| 239 | <li>xml.etree.ElementTree</li> |
|---|
| 240 | </ul> |
|---|
| 241 | </li> |
|---|
| 242 | <li>PHP</li> |
|---|
| 243 | </ul> |
|---|
| 244 | |
|---|
| 245 | <h2 id="VersionControlSVN">Version Control (SVN)</h2> |
|---|
| 246 | <p> |
|---|
| 247 | The code is in the APS version control repository: <pre>BCDAEXT</pre>. |
|---|
| 248 | <br /> |
|---|
| 249 | View the code: <a |
|---|
| 250 | href="https://subversion.xor.aps.anl.gov/trac/bcdaext/browser/pvrrd" |
|---|
| 251 | ><pre>https://subversion.xor.aps.anl.gov/trac/bcdaext/browser/pvrrd</pre></a> |
|---|
| 252 | <br /> |
|---|
| 253 | Checkout the code: svn co https://subversion.xor.aps.anl.gov/bcdaext/pvrrd/ ./pvrrd</pre> |
|---|
| 254 | </p> |
|---|
| 255 | </body> |
|---|
| 256 | </html> |
|---|
