source: pvrrd/documentation.html @ 203

Last change on this file since 203 was 203, checked in by jemian, 15 years ago

rrdcgi now enabled on WWW server -- enable live CGI scripts
  • Property svn:eol-style set to native
  • Property svn:keywords set to Date Author Revision Url Id
File size: 6.3 KB
Line 
1<html>
2<head>
3  <title>Logging EPICS PVs into RRD databases for WWW display</title>
4  <!--
5########### SVN repository information ###################
6# $Date: 2010-08-09 21:34:07 +0000 (Mon, 09 Aug 2010) $
7# $Author: jemian $
8# $Revision: 203 $
9# $HeadURL$
10# $Id: documentation.html 203 2010-08-09 21:34:07Z jemian $
11########### SVN repository information ###################
12  -->
13</head>
14<body>
15<h1>Logging EPICS PVs into RRD databases for WWW display</h1>
16<center><small>$Id: documentation.html 203 2010-08-09 21:34:07Z jemian $</small></center>
17
18 <h2> Overview </h2>
19
20These files and directories provide support for logging
21EPICS numerical PVs in RRD files and maintaining a set of
22plots for WWW browsing
23
24 <h2>Adding EPICS Process Variables</h2> -tba-
25 <h2>Updating EPICS PVs into RRD databases</h2> -tba-
26 <h2>Generating plots of logged data</h2>
27 
28     <p>
29     A WWW server properly-configured for RRDTOOL support will be
30     able to generate the plots on demand from its associated RRD database.
31     It will only generate plots as needed by the time-granularity of the plot
32     (and won't re-generate unnecessarily).
33     </p>
34
35     <p>
36     Until the XOR WWW server has rrdcgi capability, the plots and HTML pages
37     will be generated by calling rrdcgi directly from a command-line job.
38     The Python program <pre>htmlCreate.py</pre> will generate
39     all HTML pages and related plots from the set of CGI scripts.
40     It is run from a UNIX cron job.
41     </p>
42
43     <p>
44     RRDtool support was added to the WWW server in July 2010.
45     The WWW server needs to identify the CGI files by suffix.
46     Appending .cgi to all the CGI files will allow the files
47     to be properly identified and rendered.  (tested and works)
48     </p>
49
50     <p>
51     the plots directory needs to be writable by the WWW server user (nobody) and have the group
52     sticky bit set
53     </p>
54
55 <h2>Configuration of the infrastructure</h2> -tba-
56
57 <h2> crontab configuration </h2>
58
59<pre>
60        #     field          allowed values
61        #     -----          --------------
62        #     minute         0-59
63        #     hour           0-23
64        #     day of month   1-31
65        #     month          1-12 (or names, see below)
66        #     day of week    0-7 (0 or 7 is Sun, or use names)
67             
68#  log EPICS PVs into RRD for WWW browsing of logs
69# until rrdcgi can be called from WWW pages directly, regenerate WWW pages every minute
70  * * * * * /home/joule/WEB33/www/rrd/htmlCreate.py 2>&1 /home/joule/WEB33/www/rrd/htmlCreate.log
71
72# make sure that EPICS --> RRD monitor tool is running, check every 5 minutes
73# solaris cron does not support */5 notation from linux cron
74  10,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/joule/WEB33/www/rrd/checkup-epics2rrd.csh 2>&1 /dev/null
75</pre>
76
77 <h2> assets </h2>
78<pre>
79cgi/                      (ignored by SVN)
80data/                     (ignored by SVN)
81html/                     (ignored by SVN)
82other/
83plots/                    (ignored by SVN)
84checkup-epics2rrd.csh
85cgiCreate.py
86epics2rrd.csh
87epics2rrd.log
88epics2rrd.pid
89epics2rrd.py
90htmlCreate.py
91index.php
92Makefile
93makeTagHTML.py
94documentation.html
95rrdCreate.py
96xmlSupport.py
97properties.xml
98pvlist.xml
99
100leo% svn pg svn:ignore .
101xref.txt
102plots
103cgi
104html
105data
106*.log
107*.pyc
108
109</pre>
110
111<h2 id="Status">Status</h2>
112<p>
113Production version is in operation for many APS operations cycles. 
114See <a class="ext-link" href="http://sector33.xor.aps.anl.gov/rrd"
115><span class="icon">http://sector33.xor.aps.anl.gov/rrd</span></a>
116</p>
117
118<h2 id="RRDdatabasechanges">RRD database changes</h2>
119<ul>
120  <li>
121    Each EPICS PV goes into its own RRD database. 
122    This simplifies the addition of PVs into the logging system. 
123  </li>
124  <li>
125    10x finer time granularity now than before. 
126    Data are being averaged into minimum 30-second bins. 
127    (Data can always be updated to system at 1 Hz, if available.)
128  </li>
129</ul>
130
131<h2 id="DefaultWWWpage">Default WWW page</h2>
132<p>
133The default WWW page is now PHP.  This allows for greater flexibility
134in designing new interfaces.  The default interface shows a gallery of
135available PNG plots with data logged from the last week.  Each plot is
136linked to a full description of logged data (6 plots for last: hour,
137day, week, month, year, and decade).
138</p>
139
140<h2 id="Codebase">Codebase</h2>
141<p>
142Python has been used to implement the support code, calling rrdtool
143through the rrdtool Python support module.  The main WWW page is rendered in PHP.
144</p>
145
146<h3 id="Configurationdetails">Configuration details</h3>
147<p>
148Two XML files are used for configuration:
149<tt>pvlist.xml</tt> describes each of the EPICS Process Variables to be logged,
150and <tt>properties.xml</tt> describes various other configuration and implementation
151details for the support code.
152</p>
153
154<h3 id="RRDdatabasecreation">RRD database creation</h3>
155<p>
156Each PV has its own RRD database, created by <tt>rrdCreate.py</tt>.
157</p>
158
159<h3 id="CGIscriptcreation">CGI script creation</h3>
160<p>
161<tt>cgiCreate.py</tt> creates CGI scripts for each PV (and corresponding RRD database).
162</p>
163<h3 id="RRDdatabaseupdatesfromEPICS">RRD database updates from EPICS</h3>
164<p>
165The task that cmonitors EPICS PVs and enters them into RRD databases is
166called <tt>epics2rrd.py</tt>.  A <em>starter script</em> (<tt>epics2rrd.csh</tt>)
167is used to start/stop/restart the Python script,
168record the PID in a file (<tt>epics2rrd.pid</tt>),
169and pipe the output to a log file (<tt>epics2rrd.log</tt>).
170</p>
171
172<h3 id="CGIscriptoperation">CGI script operation</h3>
173<p>
174The XOR web server (<tt>leo.xor.aps.anl.gov</tt>) runs solaris. 
175The admins had difficulty compiling the various resources (cairo and pango)
176needed by the current version of rrdtools (v1.3.??)
177so <tt>rrdcgi</tt> is <strong>not available</strong> on the XOR web server now.
178</p>
179<p>
180To work around this, a crontab script runs
181<tt>web33@ipnspool3.xor.aps.anl.gov:~/www/rrd/htmlCreate.py</tt> 
182every minute and this runs each of the CGI scripts to update the plots
183as new data points become available.
184</p>
185
186<h2 id="Dependencies">Dependencies</h2>
187<ul>
188  <li>
189    Python
190        <ul>
191          <li>librrdtool</li>
192          <li>CaChannel</li>
193          <li>xml.etree.ElementTree</li>
194        </ul>
195  </li>
196  <li>PHP</li>
197</ul>
198
199<h2 id="VersionControlSVN">Version Control (SVN)</h2>
200<p>
201The code is in the APS version control repository: <pre>BCDAEXT</pre>.
202<br />
203View the code: <a 
204href="https://subversion.xor.aps.anl.gov/trac/bcdaext/browser/pvrrd"
205><pre>https://subversion.xor.aps.anl.gov/trac/bcdaext/browser/pvrrd</pre></a>
206<br />
207Checkout the code: svn co https://subversion.xor.aps.anl.gov/bcdaext/pvrrd/ ./pvrrd</pre>
208</p>
209</body>
210</html>
Note: See TracBrowser for help on using the repository browser.