Copying the data file while in use

Bill Weale (5/14/14 2:37PM)
Jeffrey Kain (5/14/14 2:43PM)
Bill Weale (5/14/14 2:56PM)
Jeffrey Kain (5/14/14 2:58PM)
Tim Nevels (5/14/14 3:04PM)
Jeffrey Kain (5/14/14 4:17PM)
Jeffrey Kain (5/14/14 5:00PM)
Joshua Fletcher (5/14/14 10:46PM)


Bill Weale (5/14/14 2:37 PM)

I guess you're asking what I'm wondering...

Will the DB engine hold strictly to your setting?

My guess is not. It depends on what the users do in your system during
the cache flush pause, but if it were me, I'd definitely want 4D to
preempt my setting if it saw the need. Which is to say that all along
I've thought that the cache flush setting was telling 4D to use that
interval as a max period between flushes, not a min.

BIll

On May 14, 2014, at 2:19 PM, Jeffrey Kain wrote:

color><param>00000,0000,DDEE/param>MMy thought was set the cache
update minutes high enough so no changes will be made to the files
while I copy them, then turn the mirror update code back on.
/color>

Jeffrey Kain (5/14/14 2:43 PM)

So far it&iacute;s holding&Ouml; I decided to try it&Ouml; :)

We have 160GB of cache memory on this server so it will take awhile
before it fills up&Ouml; It&iacute;s odd that the last modified time of the data
file matches when I did a manual FLUSH BUFFERS, but the last modified
date of the index file is about 30 minutes earlier. ?Not sure what
that&iacute;s about, but I could probably reindex the mirror pretty quickly
if it causes a problem.

Jeff

On May 14, 2014, at 2:37 PM, Bill Weale <bill.weale@...
wrote:

color><param>00000,0000,DDEE/param>II guess you're asking what I'm
wondering...

Will the DB engine hold strictly to your setting?

My guess is not. It depends on what the users do in your system during
the cache flush pause, but if it were me, I'd definitely want 4D to
preempt my setting if it saw the need. Which is to say that all along
I've thought that the cache flush setting was telling 4D to use that
interval as a max period between flushes, not a min.

BIll
/color>

Bill Weale (5/14/14 2:56 PM)

This does bring up an interesting topic comparing the risks v benefits
of a longer pause between caches. And we probably can or must consider
some 'thinking adjustments' now regarding cache flushing and the new
journaling?

I haven't looked into it, but do you know if the journal is flushed to
disk concurrent with the data?

www

On May 14, 2014, at 2:43 PM, Jeffrey Kain wrote:

color><param>00000,0000,DDEE/param>SSo far it&iacute;s holding&Ouml; I decided
to
try it&Ouml; :)

We have 160GB of cache memory on this server so it will take awhile
before it fills up&Ouml; It&iacute;s odd that the last modified time of the data
file matches when I did a manual FLUSH BUFFERS, but the last modified
date of the index file is about 30 minutes earlier. ?Not sure what
that&iacute;s about, but I could probably reindex the mirror pretty quickly
if it causes a problem.
/color>

Jeffrey Kain (5/14/14 2:58 PM)

The journal is flushed in real time so it&iacute;s always up-to-date.

On May 14, 2014, at 2:56 PM, Bill Weale <bill.weale@...
wrote:

color><param>00000,0000,DDEE/param>TThis does bring up an interesting
topic comparing the risks v benefits of a longer pause between caches.
And we probably can or must consider some 'thinking adjustments' now
regarding cache flushing and the new journaling?

I haven't looked into it, but do you know if the journal is flushed to
disk concurrent with the data?

www
/color>

Tim Nevels (5/14/14 3:04 PM)

On May 14, 2014, at 2:45 PM, Jeffrey Kain wrote:

color><param>00000,0000,DDEE/param>WWe have 160GB of cache memory on
this server so it will take awhile before it fills up&Ouml; It&iacute;s odd that
the last modified time of the data file matches when I did a manual
FLUSH BUFFERS, but the last modified date of the index file is about
30 minutes earlier. ?Not sure what that&iacute;s about, but I could
probably
reindex the mirror pretty quickly if it causes a problem.
/color>
Did I read this right -- 160GB of data cache set for 4D Server? Not
16GB but 160GB? That is incredible. I've never heard of 64bit 4D
Server running in production with a 160GB data cache. Wow!

How much RAM does this server machine have anyway? How big is the data
file that would justify a 160GB cache?

Tim

Jeffrey Kain (5/14/14 4:17 PM)

The server has 192 GB of RAM and a $40,000 PCI-E SSD array. It&iacute;s
speedy. :)

Very fast, very stable (knock on wood), running 4D Server 13.4 64-bit
and Windows Server 2008 Enterprise. The data file is currently 91GB,
indexes 27GB, about 100-110 users and 900-1100 processes. The data
file used to be quite a bit larger prior to a very large archive, but
it&iacute;s growing very rapidly.

Jeff

On May 14, 2014, at 4:04 PM, Tim Nevels <timnevels@...
wrote:

color><param>00000,0000,DDEE/param>OOn May 14, 2014, at 2:45 PM,
Jeffrey Kain wrote:

/color><color><param>8826F,0000,8219/param>WWe have 160GB of cache
memory on this server so it will take awhile before it fills up&Ouml; It&iacute;s
odd that the last modified time of the data file matches when I did a
manual FLUSH BUFFERS, but the last modified date of the index file is
about 30 minutes earlier. ?Not sure what that&iacute;s about, but I could
probably reindex the mirror pretty quickly if it causes a problem.
/color><color><param>00000,0000,DDEE/param>
Did I read this right -- 160GB of data cache set for 4D Server? Not
16GB but 160GB? That is incredible. I've never heard of 64bit 4D
Server running in production with a 160GB data cache. Wow!

How much RAM does this server machine have anyway? How big is the data
file that would justify a 160GB cache?
/color>

Jeffrey Kain (5/14/14 5:00 PM)

Yep - it ended up not working. The data file is constantly being
written to, even though it&iacute;s not &igrave;flushing&icirc; as indicated by the
flush
window on the server.

I was trying to re-synch the mirror server without bringing down
production, but that doesn&iacute;t seem possible with v13 (unless as Tom
Benedict points out I have multiple mirrors). I suspect this whole
issue would be solved with the new mirror architecture in v14.

Jeff

On May 14, 2014, at 4:46 PM, Joshua Fletcher <JFletcher@... wrote:

color><param>00000,0000,DDEE/param>BBill's right. 4D doesn't have only
one case for flushing (the interval you specify). It also flushes
whenever it needs to purge (and may flush more than once for the same
event). ?Does it flush any other time? ?I have no idea but therein
lies the risk.

You (Jeff) have two questions IMHO:

"If I copy the data file while the cache is not actively being
flushed, is the data file valid?" - If you can guarantee a flush isn't
happening, probably.

"Is this without risk?" - definitely not.

The problem is you cannot unequivocally guarantee the operation is
safe, IMO. And you risk production server instability with the copy
operation. ?I believe the copy operation will force the *system cache*
to flush, which may adversely affect system performance; at some point
access to the data file is going to be blocked.

I think some trickery with VM's or RAID would probably be a better
idea. ?A VM snapshot, for example. ?I have to imagine technology
like
this exists. ?I don't think it's as simple as doing a file copy, but
that's just suspicion.

/color><color><param>8826F,0000,8219/param>WWhat I&iacute;m trying to do is
reset the mirror server without taking
production down. A backup takes 20 minutes which isn&iacute;t possible
either (this system is 24x7).
/color>

Joshua Fletcher (5/14/14 10:46 PM)

Bill's right. 4D doesn't have only one case for flushing (the interval
you specify). It also flushes whenever it needs to purge (and may
flush more than once for the same event). ?Does it flush any other
time? ?I have no idea but therein lies the risk.

You (Jeff) have two questions IMHO:

"If I copy the data file while the cache is not actively being
flushed, is the data file valid?" - If you can guarantee a flush isn't
happening, probably.

"Is this without risk?" - definitely not.

The problem is you cannot unequivocally guarantee the operation is
safe, IMO. And you risk production server instability with the copy
operation. ?I believe the copy operation will force the *system cache*
to flush, which may adversely affect system performance; at some point
access to the data file is going to be blocked.

I think some trickery with VM's or RAID would probably be a better
idea. ?A VM snapshot, for example. ?I have to imagine technology
like
this exists. ?I don't think it's as simple as doing a file copy, but
that's just suspicion.

color><param>00000,0000,DDEE/param>WWhat I?=80&ocirc;m trying to do is
reset
the mirror server without taking
production down. A backup takes 20 minutes which isn?=80&ocirc;t possible
either (this system is 24x7).
/color>
Jeff don't forget 4D v14 "full time mirroring" already solves this for
you so I wouldn't spend TOO much effort on alternative (unsafe)
solutions. I realize there's a timeline involved in waiting for a v14
upgrade, but I didn't see it mentioned in this thread so I thought a
reminder/pointer was important.

Josh

--
Josh Fletcher
Technical Account Manager
4D, Inc

color><param>00000,0000,DDEE/param>
/color>

-----Original Message-----
color><param>00000,0000,DDEE/param>FFrom: 4d_tech-bounces@...
[mailto:4d_tech-bounces@...
On Behalf Of Bill Weale
Sent: Wednesday, May 14, 2014 11:38 AM

I guess you're asking what I'm wondering...

Will the DB engine hold strictly to your setting?

My guess is not. It depends on what the users do in your system during
the
cache flush pause, but if it were me, I'd definitely want 4D to
preempt my
setting if it saw the need. Which is to say that all along I've thought
that the cache flush setting was telling 4D to use that interval as a
max
period between flushes, not a min.

BIll
/color>

Reply to this message

Summary created 5/14/14 at 6:28PM by Intellex Corporation

Comments welcome at: feedback@intellexcorp.com