Chapter 7. The Display Modules

HTTPD comes with four small server modules for diagnosis. Three of them show live data of the running server: HTTPDSRV shows the control blocks of the server, HTTPDM shows any piece of storage as a dump, and HTTPDMTT shows the master trace table of the system. The fourth, ABEND0C1, abends on purpose to test the recovery of the server. All four are load modules that are built with the server. They are server modules in the sense of Chapter 6, “The Server Module Interface”: HTTPD runs them for the route that you give them in the Parmlib.

This chapter lists, for each module, the URL, the query parameters, what it shows, the Parmlib line, the authentication it belongs behind, and its limits. The HTTPD User’s Guide shows how an operator uses the modules, in “Operating HTTPD”.

Setting Up the Modules

HTTPD registers no route for these modules by itself. The sample Parmlib member has the four lines, commented out. They are the recommended routes:

MOD=HTTPDSRV /.dsrv     AUTH=FORM
MOD=HTTPDM   /.dm       AUTH=BASIC
MOD=HTTPDMTT /.dmtt     AUTH=BASIC
MOD=ABEND0C1 /.abend    AUTH=BASIC

The keywords of a route line are in Chapter 1, “The HTTPPRM Keywords” and Chapter 2, “Routes and Authentication Policy”. Observe these rules.

• A display module reads storage of the server address space and, for HTTPDMTT, the console log. A route without an AUTH= is public in HTTPD, so never register one of these modules without it. The sample member says the same: none of the lines may be uncommented without an AUTH= of its own.
• The modules do not look at the user. Every user who passes the route gate sees everything. To restrict the route to some users, add a RES= with a RACF profile, for example RES=FACILITY:HTTPD.ADMIN. The keyword is described in Chapter 2, “Routes and Authentication Policy”.
• HTTPDSRV writes links to the path /.dm. The route for HTTPDM must therefore be exactly /.dm if the links in the pages of HTTPDSRV are to work. HTTPDSRV is a browser tool, so AUTH=FORM (a login page) suits it. HTTPDM and HTTPDMTT also work with curl, so AUTH=BASIC suits them.
• Use the modules on a test system or for a short time. They are diagnostic tools, not production endpoints.

HTTPDSRV, HTTPDM and HTTPDMTT send the response header Cache-Control: no-store. None of them sends Access-Control-Allow-Origin, so a page of another origin cannot read an answer in a browser that holds a session with the server. The text/plain answers of HTTPDM and HTTPDMTT (d set to a number other than 0) and the pages of HTTPDSRV also send X-Content-Type-Options: nosniff, so a browser does not take the text for HTML.

A display module that is started in another way than by HTTPD (for example from JCL) does nothing. It writes the message HTTPD920E to the console and a line “This program … must be called by the HTTPD web server” to its standard output, and it ends with the return code 12.

Reading Storage

HTTPDSRV and HTTPDM read storage at addresses that come from the URL. They can read what the server task may read. They do not check an address before they read it. Four things follow from this.

• The memory dump of both modules runs under an abend handler (ESTAE). An address that the task cannot read ends the dump with a line that starts with ABEND S0C4. The server keeps working.
• The tables that HTTPDSRV prints after a dump are not under the handler. If an address in the URL of the targets FILE, TASK, MOD or WORKERS cannot be read, the module abends. HTTPD answers with the status 503 and the text “External program HTTPDSRV failed with S0C4 ABEND”, and writes the message HTTPD908E to the console. The server keeps working.
• The modules show storage as it is at the moment of the request. The values can change while the page is built.
• An address is only good while the server runs. After a restart all addresses are new.

HTTPDSRV, the Server Display

HTTPDSRV shows the control blocks of the server as HTML tables. The path in the recommended route is /.dsrv. The module answers every request method in the same way. It ignores the request body.

Parameters

The comparison of the target is not case sensitive, and the value only has to be the start of a name. HTTPDSRV tests the names in a fixed order and takes the first one that fits. The order is FILE, FS, MGR, MOD, HTTPD, TASK, TRACE, WORKERS. So target=M is MGR, target=MO is MOD, target=F is FILE, and target=T is TASK. The target TRACE needs at least the two letters TR. A value that fits none of the names gives the help page (“The Help Page”). The help page lists only the targets of this table.

Targets

Every page except the help page has a heading with the address of the block, a dump of the block with links, and a table of the fields. The table has four columns: the offset of the field in the block (in hexadecimal, with a plus sign), the name of the field, a description, and the contents.

The pages use the status 200 and the type text/html. A problem such as a missing address is reported in the text of the page, still with the status

Target HTTPD

The page shows the heading “HTTPD Handle” with the address of the block. The table that follows has one row for every field of the control block, in order of offset. The fields are these.

The page shows fields that the server no longer uses as reserved, so that the table accounts for every byte of the block. It also has a small form that posts to /.dm and shows an arbitrary memory area (“HTTPDM, the Memory Display”).

Target MOD

The target needs the address of the route array in m. The link on the field route of the HTTPD page supplies it. Without m the page says Missing memory address (&m=nnnnnnnn).

The page shows a dump of the array. For every route it shows a heading “Route #n” with MOD for a route that has a program and LOC for a static prefix, a dump of the route block, and a table:

This is the page to use to find out which policy a running server applies to a path (Chapter 2, “Routes and Authentication Policy”).

Target MGR

The target shows the thread manager of the worker pool. It does not take an address: it ignores m and uses the manager of the server. The page prints Invalid Eye Catcher if the block is not a manager. The help page says the same: the target takes no address.

The fields are the task of the manager (with a link to the target TASK), the wait flag, the function that the workers run, the user data pointer, the stack size, the worker array (with a link to the target WORKERS), the queue, the state (INIT, RUNNING, QUIESCE, STOPPED or WAITING), the minimum and maximum number of tasks, the number of requests dispatched since the start, the start time and the resource name.

Target WORKERS

The target needs the address of the worker array in m. The link on the field worker of the MGR page supplies it. The page skips an entry whose eye catcher is not a worker.

For every worker the page shows a dump and a table with the eye catcher, the wait flag, the manager, the task (with a link to the target TASK), the queue, the state (INIT, RUNNING, WAITING, DISPATCH, SHUTDOWN or STOPPED), the options (TIMER, NOWORK), the start time, the wait time and the dispatch time, and the number of requests that the worker dispatched.

When a worker is in the state RUNNING and has a queue entry, the page adds a block that tells what the worker is doing: the protocol (HTTP), the user (the user ID of the credential, or “ANONYMOUS” if there is none), the group, the socket number, and the local and the remote address with the port.

Target TRACE

The target shows the request trace that the Parmlib keyword TRACE switches on (“Diagnostics and Recording”). It needs no address. The answer is not HTML. It has the type text/plain and the header Cache-Control: no-store. If the keyword is not set, or its value was refused, the answer is the single line

#HTTPTRC off

The query parameter min is a number of milliseconds. With it, the answer leaves out each finished record whose time from the start to the last mark is shorter. A record of a request that is still running is always shown. A value that is not a number counts as 0.

The first line of the answer starts with #HTTPTRC v1 and has these words.

One line for each record follows. The lines are in the order of the table, not of time: sort by seq. A line is a row of name=value words without a space inside a value. The line of the request that reads the trace is among them, and the request is still running.

The start of a record is the accept of the connection for the first request of a connection. For a later request on the same connection it is the end of the request before, so the time that the client needs to send its next request is part of the times of that record. These are the phases, in the order a request normally reaches them.

This is the record of one request, taken from the lab server. The command puts the words three to a row; the order of the words in the line is from left to right and from top to bottom:

$ curl -s -u IBMUSER:****** "http://localhost:8080/.dsrv?target=TRACE" | grep "^seq=3 " | tr " " "\n" | xargs -n 3 printf "%-16s %-16s %s\n"
seq=3            c=1C0FC8         w=17F9C0
tcb=9BB378       ip=192.0.2.17    port=36640
n=0              resp=401         ago=1142
live=0           last=end         st=RESET
age=28           run=2            line1=3
req=6            parse=7          cred=7
route=7          lnk0=8           lnk1=27
snd1=18          sndl=21          done=27
end=28           close=-          polls=0
txn=12           txst=0           txms=0
sent=292         path=/zosmf/info

The request was the first on its connection (n=0). A module answered it with the status 401. The server read the request line 3 ms after the accept and had all header lines at 6 ms (req). The check of the policy of the route passed at 7 ms. The module was linked at 8 ms and returned at 27 ms (lnk0, lnk1). It handed its first data to TCP/IP at 18 ms and its last at 21 ms (snd1, sndl), in 12 calls of send that took together less than a millisecond. The response was complete at 27 ms, and the request was over at 28 ms.

The next example shows a connection that stays open. The command shows 11 of the words of each record. Records 3 and 4 are the two requests of one connection, and record 5 is the end of that connection. Records 1 and 2 are another connection, and record 6 is the request that reads the trace:

$ curl -s -u IBMUSER:****** "http://localhost:8080/.dsrv?target=TRACE" | tail -n +2 | cut -d" " -f1,3,7,8,10,11,13,14,16,26,27
seq=1 w=17FA40 n=0 resp=401 live=0 last=end age=60 run=6 req=16 close=- polls=0
seq=2 w=17FA40 n=1 resp=0 live=0 last=close age=3 run=0 req=3 close=3 polls=0
seq=3 w=17F9C0 n=0 resp=401 live=0 last=end age=83 run=5 req=16 close=- polls=0
seq=4 w=17F9C0 n=1 resp=401 live=0 last=end age=48 run=0 req=5 close=- polls=0
seq=5 w=17F9C0 n=2 resp=0 live=0 last=close age=508 run=0 req=507 close=508 polls=1
seq=6 w=17FB40 n=0 resp=0 live=1 last=sndl age=24 run=2 req=7 close=- polls=0

Records 3 and 4 have the same worker (w), and n counts 0 and 1. Record 4 has run=0: the worker already held the connection, so the mark is the start of the record. Record 5 has no request line (last=close, resp=0). It is the time from the end of record 4 to the close of the connection, 508 ms, and in it the server waited once for the next data (polls=1). Record 6 is still running: live=1, and its last mark so far is sndl, because the server is sending the page that shows it.

The last example shows requests whose read failed. The command shows 14 of the words of each record:

$ curl -s -u IBMUSER:****** "http://localhost:8080/.dsrv?target=TRACE" | tail -n +2 | cut -d" " -f1,3,7,8,10,11,13,14,15,16,24,25,26,27
seq=1 w=17FA40 n=0 resp=0 live=0 last=end age=5 run=3 line1=- req=5 done=- end=5 close=- polls=0
seq=2 w=17F9C0 n=0 resp=401 live=0 last=end age=25 run=2 line1=3 req=4 done=24 end=25 close=- polls=0
seq=3 w=17F9C0 n=1 resp=0 live=0 last=close age=1004 run=0 line1=- req=1003 done=- end=- close=1004 polls=2
seq=4 w=17FB40 n=0 resp=0 live=0 last=end age=9526 run=2 line1=- req=9525 done=- end=9526 close=- polls=19
seq=5 w=17FA40 n=0 resp=0 live=1 last=sndl age=18 run=1 line1=2 req=4 done=- end=- close=- polls=0

Record 1 is a connection that the client closed before it sent anything. The server stopped reading at 5 ms (req), there is no request line (line1=-), and the record ends with end. Record 2 is a request that was answered with 401, and record 3 is the end of its connection after 1,004 ms, as in the example before: the connection was kept alive and the client closed it. That record has close and no end. Record 4 is a connection that sent nothing until the server gave up after about ten seconds, the default of CLIENT_TIMEOUT. The server waited 19 times for half a second (polls=19), and the record ends with end. None of the three has done, and all have resp=0.

The times come from the TOD clock. The server shifts the clock value right by 22 bits and keeps 32 bits, so one tick is 1.024 milliseconds. Only now is in ticks. The server converts every other time of a line to real milliseconds (1000 ticks are 1024 milliseconds) and drops the fraction. The tick count wraps after about 51 days. The trace only uses the difference of two ticks, so the wrap does not matter, but the now of one readout cannot be compared with the now of a much later one.

The table does not keep the records in time order. A new record takes the next free place, counted from a position that moves by one for each request. A record of a running request is never taken. A record of a finished request that took 977 ticks (about one second) or more is taken only when no other record is left, so a slow request stays in the table longer than a fast one. The time of a request counts from the accept if it is the first request of its connection, and from its request line if it is a later one, so the time that a client takes to send its next request does not make a request slow. When all records belong to running requests, the next request gets none and lost grows.

Target TASK

The target needs the address of a thread handle in m. The fields are the eye catcher, the task control block address (tcb), owntcb, the termination ECB (termecb), the return code, the stack size, the function, its two arguments, and the address of the stack. A link on the stack opens it in HTTPDM with the stack size as the length.

Target FILE

The target needs the address of a C file handle in m. The fields are the eye catcher; the DCB address (a link opens it in HTTPDM); the assembler buffer; the logical record length and the block size; the file position; the buffer address (a link opens it) with the pointers into it; the flags, which are decoded as DYNAMIC, OPEN, READ, WRITE, APPEND, BINARY, RECORD, BSAM, TERM, ERROR and EOF the record format, which is decoded (fixed, variable or undefined, and the attributes blocked, spanned, ASA, machine control, key length); the DD name, the member, the data set name and the mode.

Target FS

Without m the page shows the UFS system block of the server: a dump of the block and the eye catcher. A line tells that the state of the disks and inodes belongs to the UFSD started task. HTTPD holds only the client side of UFS, so the page cannot show more.

Do not use m with this target. With m, the module does not display the block at that address. If the storage at the address does not start with one of the two block names that the module looks for, the module sends no response.

The Help Page

A value of target that matches no name gives the page “HTTPDSRV Help”. It has the status 200, shows the usage line with the host and the path of the request, and lists the targets. The host and the path are escaped for HTML.

HTTPDM, the Memory Display

HTTPDM shows a block of storage as a hexadecimal dump with a character column. It is the module behind the links in the pages of HTTPDSRV. It answers every request method in the same way.

Parameters

HTTPDM takes the first name that is present from each row of the table.

The rule for chunk: a number of 0 gives 16. A number above 64 gives 64. Any other number is rounded up to a multiple of 4. So 1 to 4 give 4, 5 to 8 give 8, and 61 to 64 give 64.

Output

With d=0 (the default) the response is text/html. It has a form with the fields Memory Address, Data Length and Chunk Size, and a preformatted block with the dump. With d set to another number, the response is text/plain and holds the dump only, without the form and without any report of an abend.

The dump starts with the line

Dump of [title ]AAAAAAAA (n byte chunks) (m bytes total)

in which AAAAAAAA is the address in hexadecimal, and n and m are the chunk size and the length. One line follows for every chunk. A line has the offset from the start (a plus sign and five hexadecimal digits), then the bytes in hexadecimal in groups of four, and then the characters of the bytes between colons. The character column shows the bytes as EBCDIC characters. A byte that is not a printable character shows as a period.

In the HTML form every group of four bytes is a link. The link dumps the storage to which the four bytes point, with the same length, chunk size and data parameters. The link has the form ?m=address&l=length&c=chunk&d=data. So you follow a chain of pointers with a click. In the character column the characters <, > and & are written as HTML entities.

An address that the task cannot read gives, in the HTML form, the line ABEND S0C4 occured for DISPLAY MEMORY 0x... in place of the rest of the dump. A user abend shows as ABEND Unnnn occured.... If the abend handler cannot be created, the line is ESTAE CREATE failure, RC=0x... with the code. With d other than 0 these lines are not written and the output simply stops.

Limits

• The largest dump is 4096 bytes per request.
• HTTPDM keeps the low 24 bits of the address, because the system uses 24-bit addresses. The form always shows the address that the module used.
• The route belongs behind a login (“Setting Up the Modules”), because the dump shows storage of the server.
• The dump can reach any storage of the server address space that the task may read. That may include data that the server holds for its users, such as cached credentials.

HTTPDMTT, the Master Trace Table

HTTPDMTT shows the master trace table of the system. The table holds the recent messages of the system console. The module makes a copy of the table and prints the entries in order.

Parameters

Output

With d=0 the response is text/html. The page has the heading “Master Trace Table”, then one preformatted block with one line for every entry of the table. With d set to another number, the response is text/plain and has the lines only. The module prints the text of an entry as text. It does not read the text as a format, so a percent sign in a message does no harm.

In the HTML page the module escapes <, >, &, " and ' in every entry, so a message such as <b> shows as that text and is not read as markup. The table holds the messages of every job, not only those of HTTPD. In the plain form the entries are as the table holds them, without escaping. The module reads an entry up to its length in the table, or up to a null character if that comes first.

If the module cannot get storage for the copy, it prints “Unable to allocate Master Trace Table.” and ends. The module frees the copy at the end.

The module sends an entry in pieces of at most 255 characters, after any escaping. It checks the sends of the response header, of the start of the HTML page up to the body tag, and of the entries. At the first of these that fails, for example because the client has closed the connection, it stops and sends nothing more: not the rest of the entry, not the next entries and not the closing lines of the HTML page. An entry longer than one piece can therefore end in the middle of the line. The module frees the copy in this case too. It does not check the sends of the heading, of the opening line of the preformatted block, of the closing lines of the page and of the allocation message; a failure there does not stop it.

Limits

• The table is the master trace table of the system, so it shows every message that went to the console. It exposes the console log to everyone who may use the route. It needs strong authentication.
• The module does not select, filter or search. Take the plain form (d=1) and use a tool of your own for that.

ABEND0C1, the Abend Test

ABEND0C1 takes storage and abends with a program check (S0C1). It exists to prove that HTTPD gives back the storage of a failed module and keeps its workers (“Abends”). Call it many times and check that the other modules still load and run as fast as before.

The module looks for the text kb= anywhere in the query string, not case sensitive, and reads the digits after it. An example is /.abend?kb=512.

What the module does:

1. It takes the storage in blocks of 16 KB with malloc and writes the byte hex C1 into every block. It stops when no storage is left.
2. It writes the console message HTTPD921W. The message names the amount taken in KB and the number of blocks.
3. It abends with S0C1, without freeing the blocks. It sends no response itself.

The client then gets the answer of HTTPD for a failed module: the status 503 with the text “External program ABEND0C1 failed with S0C1 ABEND”. HTTPD releases the storage that the module took (“Abends”).

The default amount does not endanger the server. The module is a probe that you may point at a running system.