intermittent stalling in web application

Rebecca Bryant (5/29/14 8:33AM)
Rebecca Bryant (5/29/14 2:57PM)
Rebecca Bryant (5/29/14 5:26PM)
Timothy Penner (5/29/14 9:05PM)
Joshua Fletcher (5/29/14 10:23PM)
Keisuke Miyako (5/30/14 1:00AM)
Joshua Fletcher (5/30/14 1:25AM)
Keisuke Miyako (5/30/14 4:09AM)

Rebecca Bryant (5/29/14 8:33 AM)

<CAO6o6=Vj+JY6xwOBT2gW0cAs253Mgmqic4jDcrAMBJW7AtsETg@mail.gmail.com>

I am trying to optimize performance of a 4D web application/database
with a
large data file (millions of records). The application intermittently
stalls when several users are performing a search and the application
also
intermittently displays the 'Not enough stack space to complete the
current
method' dialog. ?Researching knowledgebase I found this note from a
January
response from Josh Fletcher to a developer having similar issues: "
...It's
unlikely memory is the cause of your slow down. More likely candidates
are
a pegged CPU core (since this is a Web app, all 4D access is
single-threaded) or the disk." Could someone please
translate/elaborate on
the following for me?

1) pegged CPU core
2) since this is a Web app, all 4D access is single-threaded
3) or the disk

Thank you!
Rebecca Bryant
--
beccabry@...
Rebecca Bryant (5/29/14 2:57 PM)

<CAO6o6=X4Bm-akzYKyBCLmSnXOxmj33oAnc0C8mh9so0mh3BqOg@mail.gmail.com>

To Balinder,

So sorry. I meant to include these specifics and forgot.:

Windows Server 2008 R2 Standard
64-bit operating system
10.0 GB RAM
Intel Xeon CPU ?2.53 GHz

4D v12.4

To Josh ( and anyone else willing to share his/her knowledge): Thanks
for
the info. Some stupid questions:

(1) "Either your stack size is too small to handle these requests, or
you
have an infinite recursion. "

Where exactly would you set stack size in a web application? I'm not
explicitly creating a new process for each web request because I
thought 4D
creates a new process for each web request. I use 4dcgi so everything
hits
OWA first and handle requests there and use OWC mostly to send the page
(although there is some handling in OWC based on what page is being
sent).
I think that is the opposite of the way others handle it?

Also, another question: When watching application activity via Runtime
Explorer sometimes just 'WebProcess' appears/clears and sometimes a
specific method call appears/clears. What are the 'WebProcess'
entries? Can
you point me toward some texts or other references that would help me
understand all of this more clearly?

(2) "You have 4D code in your database. ?All 4D code is single
threaded in
the host session....

I understand the basic concept of single-threading. I.e., the requests
basically wait in line for time in the CPU instead of the requests
being
distributed among multiple cores. I don't understand what you mean by
"single threaded in the host session". What is the host session? Is
that
just a term for the overall "connection" instance between the
application
and cyberspace while the application is up? Again, can you point me
toward
some references? (I tried Googling host session but got links
discussing
'session host'.) Are there other tools for building web applications
that
are not single-threaded in the host session? Or if the front end and
back
end are separate entities is that one way to achieve mult-threading?

(3) "Therefore all HTTP requests that pass through On Web Connection
are
also single-threaded (because they are executing 4D code). ?Therefore
it's
relatively easy for a Web application to saturate a single CPU core if
the
requests are complex enough."

Or if the requests are numerous/frequent enough? (1) Can you point me
toward some references for optimizing 4D's handling of requests? (2)
Are
there tools better suited (than 4D) for front-ending very large
databases
or systems that have lots of activity (searching, generating PDFs)?

And any thoughts on optimizing name searches? Those are the ones that
take
the longest. Also, we run interpreted so we can make changes
quickly/easily. Would running compiled speed up queries or only code
execution?

To Jeff Kain if you're listening. You've been mentioned as someone who
regularly works with large databases. Are any of yours front-ended by
a web
portal or are they all client/server?

Also, one thing I discovered by examining the logs I keep is that what
often stalls the application is when users click the Search button
multiple
times because the results are not returned quickly enough. One user
clicked
'Search' 14 times within one second. This caused a search of over 3
million
records 14 times and pretty much brought everything to a screeching
halt.
We're working with the users to help them understand that they should
not
do this but it is a very common human response. Like hitting the
elevator
button over and over as though that will make it arrive at your floor
more
quickly.

color><param>00000,0000,DDEE/param>II am trying to optimize
performance of a 4D web application/database with
/color>aa
color><param>00000,0000,DDEE/param>llarge data file (millions of
records). The application intermittently
stalls
/color>
Thank you!
Rebecca Bryant
beccabry@...
Rebecca Bryant (5/29/14 5:26 PM)

<CAO6o6=Wmd9AqogT2z4UzpCygiVFKBxb9sJN8Wq5fQF4UKvWn2w@mail.gmail.com>

Timothy - Thank you very much for the suggestion to disable the Search
button.

Josh -- Thank you so much for your answers. Very, very helpful. Some
notes/add'l questions:

color><param>00000,0000,DDEE/param>OOWA is for authentication, not
request handling. ?I have a feeling there's
/color>ssome context missing here

Yes, I misspoke about OWA. I authenticate there and then pass the
request
on for processing.

color><param>8826F,0000,8219/param>WWhat are the 'WebProcess' entries?
/color><color><param>00000,0000,DDEE/param>AAre you referring
specifically to the process names?
/color>
No, they just say "Web Process". (I typed it incorrectly before; there
is a
space between the words.)

color><param>00000,0000,DDEE/param>TTry this: build yourself a small
database, turn on the Web server, build
/color>ssome pages that use each of the 4D tags you're using, and
trace each call.
Trace OWA, OWC, the method calls, etc.

This is in fact how I taught myself to build 4D web applications, using
David Adams book (and much of his generously shared code).

color><param>00000,0000,DDEE/param>....Node is single-threaded to test
the theory that it's better for most
/color>WWeb-based activities.

What is the basic premise of this theory? Perhaps because
record-locking is
hard to achieve effectively on the web?

color><param>00000,0000,DDEE/param>II guess another way to say it is
if there's a single request that takes a
/color>llong time to execute that's not a "stall". You asked the
server to do
something that takes a long time. ?To me "stalling" means that my
request
is waiting because someone else's request is running.

Exactly. Other requests are having to wait because of the user who
clicked
the Search button 14 times, for example. So it's not the initial query
that
I'm referring to as "stalled". Also, the application only seizes up
when
several of these queries stack up.

I am confused about one thing though. If I watch Runtime Explorer
while a
slow query is executing (the name search seems to be the only truly
concerning query in terms of speed) ?I will see other processes (not
the
mysterious (to me) 'Web Process' but requests for this or that page, --
like clicking on a link in a list of records to drill down into a
specfic
record for example) I'll see other requests pop up in Runtime Explorer
and
then disappear. Which I assume to mean the page requested was sent
back to
the browser. So these requests are not having to wait for the query to
finish. Is 4D alternating between the query and letting other requests
have
a little CPU time instead of having to wait their turn? Thanks for the
'Thread' link. I thought I understood the concept but don't understand
how
these other requests can get granted while the query is going on. I'm
glad
it works this way; just don't understand yet.

color><param>8826F,0000,8219/param>((2) Are
there tools better suited (than 4D) for front-ending very large
databases
or systems that have lots of activity (searching, generating PDFs)?
/color>
color><param>00000,0000,DDEE/param>AAre you referring to 4D as
middleware? I.e. the data is not stored in 4D?
/color>
Yes, that's what I was referring to.

color><param>00000,0000,DDEE/param>IIf the data's already in 4D I
think you'd be hard-pressed to find
/color>ssomething "better". By that I mean if the data set is small
enough to fit
in a 4D application, 4D's is pretty damn good at managing it.... For
the
moment, based on what you've posted so far, I don't think you're in a
situation where you're hitting any 4D limitation...

Good to know! Thank you.

color><param>8826F,0000,8219/param>AAnd any thoughts on optimizing
name searches? Those are the ones that
/color>ttake
color><param>8826F,0000,8219/param>tthe longest.
/color>
color><param>00000,0000,DDEE/param>IIndexing is the most important
optimization. Assuming the fields are
/color>aalready indexed, it's unlikely an indexed query even over
millions of
records would be perceivable to the user. There's probably something
more
complex going on.

Yes, the field is indexed and yes, it is a bit more complex. Have to
chew
on this a bit.... I might post another thread.

color><param>00000,0000,DDEE/param>TThe first optimization you should
make is running compiled.
/color>
So running compiled DOES actually speed up queries?

color><param>00000,0000,DDEE/param>AAs Tim mentioned disabling the
search button is a good first step, ...The
/color>uuser is authenticated, right (so you know who they are)? When
>the user
makes a search request, make sure they don't already have one running.
?If
they do, prevent them from executing additional >searches. ...

Excellent idea! Thank you.

Your time and help are very much appreciated Josh!

Rebecca Bryant
--
beccabry@...
Timothy Penner (5/29/14 9:05 PM)

I have seen e-commerce sites that will disable the submit button on
the checkout page to prevent multiple charges to a your credit card; I
suppose you could do the same thing; disable the search button via
javascript upon being clicked?

-Tim

-----Original Message-----
Also, one thing I discovered by examining the logs I keep is that what
often stalls the application is when users click the Search button
multiple times because the results are not returned quickly enough.
One user clicked 'Search' 14 times within one second. This caused a
search of over 3 million records 14 times and pretty much brought
everything to a screeching halt.
We're working with the users to help them understand that they should
not do this but it is a very common human response. Like hitting the
elevator button over and over as though that will make it arrive at
your floor more quickly.
Joshua Fletcher (5/29/14 10:23 PM)

Hi there.

color><param>00000,0000,DDEE/param>
/color>

-----Original Message-----
color><param>00000,0000,DDEE/param>WWhere exactly would you set stack
size in a web application?
/color>
Sorry it was an oversight on my part, I should have reversed those
suggestions. ?There was a mechanism to change internal stack sizes
using Resources but I'm not sure it works anymore and it's really rare
to run out of stack for the kind of tasks a Web process would handle.
?An infinite recursion is more likely IMO.

color><param>00000,0000,DDEE/param>II use 4dcgi so everything hits
OWA first and handle requests there and use OWC mostly to send the page
(although there is some handling in OWC based on what page is being
sent).
I think that is the opposite of the way others handle it?
/color>
OWA is for authentication, not request handling. ?I have a feeling
there's some context missing here because this doesn't really make
sense to me. Are you saying you're running business logic from OWA
other than authentication? ?If so that's something to consider
changing.

color><param>00000,0000,DDEE/param>AAlso, another question: When
watching application activity via Runtime
Explorer sometimes just 'WebProcess' appears/clears and sometimes a
specific method call appears/clears. What are the 'WebProcess' entries?
Can
you point me toward some texts or other references that would help me
understand all of this more clearly?
/color>
Are you referring specifically to the process names? IMO the best way
to learn is to trace them so you know what they're doing.

Try this: build yourself a small database, turn on the Web server,
build some pages that use each of the 4D tags you're using, and trace
each call. ?Trace OWA, OWC, the method calls, etc. ?Note the process
names when you trace. ?It's important to gain first-hand experience of
this kind of stuff when developing Web apps IMO.

color><param>00000,0000,DDEE/param>WWhat is the host session?
/color>
Sorry I shouldn't have chosen the word "session" in a discussion about
Web apps, that's confusing. ?When I use the word "session" I most
often mean "instance of the software". I mean the instance of the 4D
software. The distinction that I was trying to allude to is that 4D
Server supports pre-emptive execution of some parts of 4D. But not "4D
language". ?Any literal 4D code running in 4D Server is
single-threaded (e.g. a stored procedure, a Web process, or a method
with the "Execute on server" attribute). Point of clarification for 4D
Web apps: static requests do not manifest as 4D processes so you know
that any 4D process you see is a dynamic request.

color><param>00000,0000,DDEE/param>AAre there other tools for building
web applications that
are not single-threaded in the host session?
/color>
Most definitely. Probably every modern web stack uses a multi-threaded
programming language though there are some single-threaded ones now
too (like Node). But 4D is single-threaded for simplicity and because
of the language legacy, whereas Node is single-threaded to test the
theory that it's better for most Web-based activities.

color><param>00000,0000,DDEE/param>OOr if the front end and back
end are separate entities is that one way to achieve mult-threading?
/color>
Multi-threading in this context refers to the ability of a single
instance of the software to execute tasks using more than one CPU core
(or, more technically, the ability of the OS scheduler to split the
tasks up). ?"Thread" is a technical term:

http://en.wikipedia.org/wiki/Thread_(computing)

color><param>00000,0000,DDEE/param>OOr if the requests are
numerous/frequent enough?
/color>
To me it's not "or", it's both. ?You need both to see a problem. ?I
guess another way to say it is if there's a single request that takes
a long time to execute that's not a "stall". You asked the server to
do something that takes a long time. ?To me "stalling" means that my
request is waiting because someone else's request is running.

color><param>00000,0000,DDEE/param>((1) Can you point me
toward some references for optimizing 4D's handling of requests?
/color>
Posting here is a great place to start since your peers will have the
best practical experience.

Also use http://kb.4d.com (if you haven't already). There are quite a
few assets about Web technology and 4D as well as assets about
optimization. Preemptive note: when I say "use KB" I am assuming you
have full access to everything on KB; that is to say some KB content
is limited (to 4D Partners, for example). So if you don't have access
to something just be aware it's outside my abilities to supplement the
content that's already available on KB.

color><param>00000,0000,DDEE/param>((2) Are
there tools better suited (than 4D) for front-ending very large
databases
or systems that have lots of activity (searching, generating PDFs)?
/color>
Are you referring to 4D as middleware? I.e. the data is not stored in
4D? Certainly there are tools that are "faster" or can scale larger
than 4D. But that doesn't make them "better". Usually they are more
complex and more expensive. 4D does a lot of things *really well* but
it's not a panacea.

If the data's already in 4D I think you'd be hard-pressed to find
something "better". By that I mean if the data set is small enough to
fit in a 4D application, 4D's is pretty damn good at managing it. ?If
the data set is large enough to require something like map-reduce
(i.e. data distributed over hundreds or thousands of nodes) you
probably wouldn't be using 4D anyway.

For the moment, based on what you've posted so far, I don't think
you're in a situation where you're hitting any 4D limitation...

color><param>00000,0000,DDEE/param>AAnd any thoughts on optimizing
name searches? Those are the ones that take
the longest.
/color>
Indexing is the most important optimization. Assuming the fields are
already indexed, it's unlikely an indexed query even over millions of
records would be perceivable to the user. There's probably something
more complex going on. Have you looked at the Query Paths to make sure
you're not running sequential searches, for example?

color><param>00000,0000,DDEE/param>AAlso, we run interpreted so we can
make changes
quickly/easily.
/color>
Making changes in production systems is dangerous. I would advise
against this in all situations. Making updates quickly is a matter of
automating deployment. It could be as simple as a couple of batch
files and code to automate the process of compiling and deploying the
app. ?It's not a huge investment to achieve this automation IMO and it
has a huge payoff.

color><param>00000,0000,DDEE/param>WWould running compiled speed up
queries or only code
execution?
/color>
The first optimization you should make is running compiled.

color><param>00000,0000,DDEE/param>AAlso, one thing I discovered by
examining the logs I keep is that what
often stalls the application is when users click the Search button
multiple
times because the results are not returned quickly enough. One user
clicked
'Search' 14 times within one second. This caused a search of over 3
million
records 14 times and pretty much brought everything to a screeching
halt.
We're working with the users to help them understand that they should
not
do this but it is a very common human response. Like hitting the
elevator
button over and over as though that will make it arrive at your floor
more
quickly.
/color>
As Tim mentioned disabling the search button is a good first step, but
it won't stop the user from refreshing the page. You should deal with
this on the server, IMO. The user is authenticated, right (so you know
who they are)? When the user makes a search request, make sure they
don't already have one running. ?If they do, prevent them from
executing additional searches. You should do this anyway because the
current design is vulnerable to DOS attack.

Good luck!

-Josh
Keisuke Miyako (5/30/14 1:00 AM)

I would hazard a guess that it is the query code not taking full
advantage
of the 4D language.

"millions of records?=A2?=82¨=C2&ugrave; is a very small database in 4D
and should take just a fraction of a second to complete a search,
even in version 2004.

besides, you shouldn't need a large process stack to perform a query,
unless you store large (intermediate) data sets in local variables or
use
a plugin;
the query should be written in a way that takes advantage of the main
cache which is shared between processes.

same with CPU core; the query is multi-threaded (except in a single
user
4D running as a web server)

I think it would be interesting to stress test the query logic directly
from a 4D method
outside a web context and analyze its performance (debug log, query
path).

miyako

On 2014/05/29 21:33, "Rebecca Bryant" <beccabry@... wrote:

color><param>00000,0000,DDEE/param>11) pegged CPU core
2) since this is a Web app, all 4D access is single-threaded
3) or the disk
/color>
Joshua Fletcher (5/30/14 1:25 AM)

color><param>00000,0000,DDEE/param>
/color>

-----Original Message-----
color><param>00000,0000,DDEE/param>WWhat is the basic premise of this
theory? Perhaps because record-locking
is
hard to achieve effectively on the web?
/color>
No it's not about database concepts at all. ¬=A0It's about resource
efficiency and ease of programming. ¬=A0I don't even want to attempt
an
explanation, there's some really good ones out there already :)
Don?=A2?=82¨=E2&Ntilde;¢t confuse single-threaded with synchronous
though, it's not the
same thing (see next part)...

color><param>00000,0000,DDEE/param>II am confused about one thing
though. If I watch Runtime Explorer while a
slow query is executing (the name search seems to be the only truly
concerning query in terms of speed) ¬=A0I will see other processes
(not
the
mysterious (to me) 'Web Process' but requests for this or that page, --
like clicking on a link in a list of records to drill down into a
specfic
record for example) I'll see other requests pop up in Runtime Explorer
and
then disappear. Which I assume to mean the page requested was sent
back to
the browser. So these requests are not having to wait for the query to
finish. Is 4D alternating between the query and letting other requests
have
a little CPU time instead of having to wait their turn?
/color>
Of course. Single-threaded doesn't mean fully synchronous. 4D has an
internal "scheduler", written before the days of multi-core
processors, to provide "cooperative multi-tasking":

http://kb.4d.com/assetid=75963

When the user clicked the Search button 14 times, it created 14 4D
processes. ¬=A0The Scheduler gives each of those processes a little
slice
of time to execute but, as you said, those queries are slow so they
probably need more than 1 slice to finish, whereas some other request
coming in might have no problem completing within 1 slice.

Given your example above, you've exposed another possible explanation
for the "stalls": the stalled requests are trying to access a resource
that is not sharable (e.g. locked record). They have no choice but to
wait until the first request is complete (the record is no longer
locked). ¬=A0But you would probably already know this (if the
processes
were waiting for a record to be unlocked you'd have written code to
manage that).

I suppose it's important to make a distinction between "stalled" and
"slow". ¬=A0To me "stalled" means "waiting" and not "slow". ¬=A0I
realize
that the user perception is that it's "stalled" because the browser
isn't responding but you can't think like a user here of course. It's
important to get a handle on what those processes are doing. Example:
instead of clicking the search button 14 times from the same browser
tab, make 14 searches from different browser tabs and see if they
eventually all get their results at roughly the same time, or do the
results come back in series. This would tell you if it's just a matter
of the fact that the search takes a long time, or rather that you can
only do 1 search at a time of this type.

color><param>00000,0000,DDEE/param>TThanks for the
'Thread' link. I thought I understood the concept but don't understand
how
these other requests can get granted while the query is going on. I'm
glad
it works this way; just don't understand yet.
/color>
I think one key thing to understand is it's the OS who decides how
things are executed in a multi-threaded environment, not the app. ¬=A0Th
e
app just says, "here are my distinct tasks, please execute them; note
they don't depend on each other". The OS takes those requests and
spreads them out over the CPU cores.

You intuitive understanding is correct: you can't do more than one
thing at the same time in any single thread. ¬=A0But if one thread is
dedicated only to receiving requests and then it forwards those
requests to other threads, you can make it *look like* there's more
than one thing happening at the same time.

color><param>00000,0000,DDEE/param>SSo running compiled DOES actually
speed up queries?
/color>
I think about it differently: don't even start optimizing anything
until you test compiled first. ¬=A0That's the first step.

Kind regards,

Josh

--
Josh Fletcher
Technical Account Manager
4D, Inc
Keisuke Miyako (5/30/14 4:09 AM)

I posted an incorrect piece of information a few moments ago:

wrong: the query is multi-threaded
correct: the query is multi-threaded if called in a pre-emptive thread

the confusion came from the famous ?=A2?=82¨=C5&igrave;fantastic
four?=A2?=82¨=C2&ugrave; diagram,
describing which part of 4D is multi-threaded,
and which part is single-threaded.

It said the the database ?=A2?=82¨=C5&igrave;engine" is multi-threaded,
but maybe it should have said that the database engine was ?=A2?=82¨
?=93thread
safe?=A2?=82¨=C2&ugrave;.

to say that the engine is multi-threaded might give the false
impression
that

there is a database server running intrinsically multi threaded on 4D
Server,
but in reality the ?=A2?=82¨=C5&igrave;engine" itself isn't a service,
it's an API.

it is safe to run in a multi threaded context,
but in 4D such a context can only be created over the network,
thru ports 19812 or 19814.

so if a stored procedure (and all web processes are stored procedures)
runs a QUERY on the server,
it is single threaded from start to finish because the context is a
co-operative thread.

on the other hand, if a 4D Remote calls QUERY on the client,
then, the search is multi-threaded because the context is a pre-emptive
thread.

you might recall an old discussion on ?=A2?=82¨=C5&igrave;which 4D
commands are
pre-emptive?=A2?=82¨=C2&ugrave;,
and you heard names like SAVE RECORD, ORDER BY, QUERY (not BY ARRAY),
etc.
but all of that was referring to commands called on the client,
not in a stored procedure or a web process.

anyway,
my post must have been contradictory to information coming from
reliable
sources,
I sincerely apologize for any confusion caused.

keisuke miyako
Reply to this message

Summary created 5/30/14 at 9:17AM by Intellex Corporation

Comments welcome at: feedback@intellexcorp.com