Samba question

[dudeonthebayou]

I have installed PCLinuxos 2012 on a old Dell Poweredge 1800, i installed all the updates through Synaptic and configured Samba without any tweeks. I can browse the system over the network from my xp pro workstation and I can read and write to the shares. The issue that I have is the transfer speed. When I upload a file (i tried a 2GB PST file) to the samba server i get between 6-10MB/s but when I download to the xp workstation i get 45-50MB/s. I can upload to a windows 2003 server (2GB about 45 seconds) while uploading to samba about 6 minutes. Is there anything I can change or tweek to boost the upload speed? Also it does seem to hesitate alot while uploading to the samba server even when uploading multiple small files as well. It almost seems like a buffer issue to me. Any pointers would be appreciated.

[muungwana]

To a hard drive,writes are more expensive than reads.

You are writing to an old computer,maybe the write speed of the hard drive is very low and its the cause of the issue.

If you,put the 2GB file in a usb stick or something else and then transfer copy it over to the computer and note the copy speed.
Saying the above differently,try copying a large file using other means to the computer and note the time it will take to finish the transfer.

Comparing the two ways will shed more light on where the bottleneck could be

[dudeonthebayou]

I tested FTP transfer and that was slower than smb transfer. I really wonder if its controller card problem since its connected to a perc5 card and its a brand new harddrive.

[muungwana]

you can try direct write to the drive and see how it will perform.

open the terminal and run this command and post its output here.

The command will create a 200M file and your hard drive may complain a bit

dd if=/dev/zero of=test.img bs=200M count=1

[pags]

you can try direct write to the drive and see how it will perform.

open the terminal and run this command and post its output here.

The command will create a 200M file and your hard drive may complain a bit

dd if=/dev/zero of=test.img bs=200M count=1

You can try changing the command to:

Code: [Select]

dd if=/dev/urandom of=test.img bs=200M count=1

That will reduce the chance of buffering (continual zeroes) making the performance look better than it is...

[dudeonthebayou]

dd if=/dev/urandom of=test.img bs=200M count=1

gives me 209715200 bytes (210MB) copied, 53.2123 s 3.9 MB/s.

This is a brand new WD 1TB Sata harddrive, its pulled from another system to use just for testing.

[pags]

Code: [Select]

dd if=/dev/urandom of=test.img bs=200M count=1
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 21.4671 s, 9.8 MB/s
This is on a laptop...

Code: [Select]

dd if=/dev/urandom of=test.img bs=200M count=1
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 33.4363 s, 6.3 MB/s
This is on an older desktop with an unexceptional SATA drive...

I would expect you'd want to see better numbers on yours...

[zorlac]

My laptop just now.

$ dd if=/dev/urandom of=test.img bs=200M count=1
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 47.4561 s, 4.4 MB/s

[muungwana]

you should run the command with "/dev/zero" device.The speeds you guys are getting are a reflection of CPU power and not hard drive write through.

The idea is to push through as much data as possible to the hard drive and see how fast it will be able to write it.

Writing 4.4MB per seconds hardly tests the hard drive.

[zorlac]

you should run the command with "/dev/zero" device.The speeds you guys are getting are a reflection of CPU power and not hard drive write through.

The idea is to push through as much data as possible to the hard drive and see how fast it will be able to write it.

Writing 4.4MB per seconds hardly tests the hard drive.

Yep, this looks a lot more accurate.

dd if=/dev/zero of=test.img bs=200M count=1
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 1.74696 s, 120 MB/s

How would I direct this test to other drives & partitions?
Thanks.

[pags]

you should run the command with "/dev/zero" device.The speeds you guys are getting are a reflection of CPU power and not hard drive write through.

The idea is to push through as much data as possible to the hard drive and see how fast it will be able to write it.

Writing 4.4MB per seconds hardly tests the hard drive.

Well, yes and no...any moderately modern CPU isn't going to have a lot of trouble generating psuedo-random values, and real world data is rarely a single contiguous value.

Using /dev/zero allows the PC to buffer and modify the data more efficiently, releasing the process before it actually done.

However, I agree that real-world data is not random, either (there are parts of the stream that will be repetitive. The actual value is probably between the two...

Just remembered hdparm:

Code: [Select]

su -c "hdparm -tT /dev/sda"
Password:

/dev/sda:
 Timing cached reads:   11884 MB in  1.99 seconds = 5965.64 MB/sec
 Timing buffered disk reads: 202 MB in  3.02 seconds =  66.94 MB/sec

This the same laptop that gave me 9.8, earlier.  Look at the second number (~67).
Compare to this:

Code: [Select]

dd if=/dev/zero of=test.img bs=200M count=1
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 0.358604 s, 585 MB/s

There's no way this thing could handle writing half a gig per second...


BTW, that hdparm command needs root privileges, which is what the "su -c" is for.  You'll need to supply the root password when prompted.

[muungwana]

Below is a bit more accurate test.

to run the test,copy the source code into a text file,name the text file "speedTest.c" and then compile it with the below command to create a program named "speedTest"

Code: [Select]

gcc -o speedTest speedTest.c -lrt

Run the program with a 200MB( or any large enough) file and check the generated output.The program take two arguments,the first one is the path to the file to use as input and the second one is the path to where the copied file will be stored.Careful about the second argument as it will overwrite a file if the path is already taken

example use of the program

[ink@mtz ~]$ ./speedTest test.img test1.img
data written: 200.00 MB
time taken: 7.54 seconds
average rate:26.51 MB/s
[ink@mtz ~]$

Code: [Select]


#include<stdio.h>
#include<time.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

int main( int argc,char * argv[] )
{
if( argc != 3 ){
return printf( "argument list != 3\n" ) ;
}

const char * file1 = argv[ 1 ] ;
const char * file2 = argv[ 2 ] ;

int fd1 = open( file1,O_RDONLY ) ;
if( fd1 == -1 ){
return printf( "could not open source file for reading\n" ) ;
}

int fd2 = open( file2,O_WRONLY | O_TRUNC | O_CREAT ) ;
if( fd2 == -1 ){
close( fd1 ) ;
return printf( "could not open destination file for writing\n" ) ;
}

mode_t mode = S_IRWXU | S_IRGRP | S_IXGRP | S_IXOTH | S_IROTH ;

fchmod( fd2,mode ) ;

struct stat st ;
fstat( fd1,&st ) ;

struct timespec start ;
struct timespec end ;

void * map = mmap( 0,st.st_size,PROT_READ,MAP_PRIVATE,fd1,0 ) ;

if( map != MAP_FAILED ){

clock_gettime( CLOCK_REALTIME,&start );
write( fd2,map,st.st_size ) ;
clock_gettime( CLOCK_REALTIME,&end );

float time = ( end.tv_sec - start.tv_sec ) + ( ( end.tv_nsec - start.tv_nsec ) * 0.000000001 ) ;

float rate = ( float )st.st_size/ time ;

rate = rate / ( 1024 * 1024 ) ;

printf( "data written: %.2f MB\n",( float )st.st_size / ( 1024 * 1024 ) ) ;
printf( "time taken: %.2f seconds\n",time ) ;
printf( "average rate:%.2f MB/s\n",rate ) ;
munmap( map,st.st_size ) ;
}else{
printf( "test failed\n" ) ;
}

close( fd1 ) ;
close( fd2 ) ;
return 0 ;
}


[pags]

Cool.  Thanks.

Out of curiosity, I tried it with a file made of zeroes:

Code: [Select]

./speedtest test.img test1.img
data written: 200.00 MB
time taken: 0.30 seconds
average rate:656.45 MB/s

And a file with random data:

Code: [Select]

./speedtest testu.img testu1.img
data written: 200.00 MB
time taken: 0.58 seconds
average rate:345.70 MB/s

I think these rates are still kind of optimistic (but I don't know C well enough to be critical of the process), but my point is still the type of data being written has an impact on performance.

Thanks.

[Just17]

Would using the

fdatasync  or fsync
             

option with dd give more accurate results?


dd  if=/dev/zero of=/home/user/dd-test  bs=200M count=1 conv=fsync


EDIT:

My results on a 4 core CPU ......  as I have always found in the past using urandom is very heavy (time wise) and delays dd greatly.
I had tried using it for wiping HDDs but had to give it up due to the slow down in the process.

dd  if=/dev/zero of=/home/user/dd-test  bs=200M count=1 conv=fsync
209715200 bytes (210 MB) copied, 3.05924 s, 68.6 MB/s

dd  if=/dev/zero of=/home/user/dd-test  bs=200M count=1
209715200 bytes (210 MB) copied, 2.663 s, 78.8 MB/s


dd  if=/dev/urandom of=/home/user/dd-test  bs=200M count=1 conv=fsync
209715200 bytes (210 MB) copied, 28.4451 s, 7.4 MB/s

dd  if=/dev/urandom of=/home/user/dd-test  bs=200M count=1
209715200 bytes (210 MB) copied, 28.1501 s, 7.4 MB/s

[muungwana]

Cool.  Thanks.

Out of curiosity, I tried it with a file made of zeroes:

Code: [Select]

./speedtest test.img test1.img
data written: 200.00 MB
time taken: 0.30 seconds
average rate:656.45 MB/s

And a file with random data:

Code: [Select]

./speedtest testu.img testu1.img
data written: 200.00 MB
time taken: 0.58 seconds
average rate:345.70 MB/s

I think these rates are still kind of optimistic (but I don't know C well enough to be critical of the process), but my point is still the type of data being written has an impact on performance.

Thanks.

I think this is the best test you can have since all it does is copy a file from memory to disk and hence the speed you will get is a reflection only of the capacity of the hard drive to take it data.The test is equivalent to download 200MB of data into memory and then commit all of it to disk with a single command.



The code is pretty simple,this is how it works

int fd1 = open( file1,O_RDONLY ) ; //<----- 1
int fd2 = open( file2,O_RDONLY ) ; //<-------2
void * map = mmap( 0,st.st_size,PROT_READ,MAP_PRIVATE,fd1,0 ) ;//<-------3
clock_gettime( CLOCK_REALTIME,&start );//<------4
write( fd2,map,st.st_size ) ;//<-------5
clock_gettime( CLOCK_REALTIME,&end );//<--------6

The first command opens the source file

The second one opens the destination file

The third commands maps the source file into memory to allow file access as if it was simple memory access.Possible alternative to mapping the file is to read the whole of it into memory first,or read a bit of it or source file into memory and write it to destination file in a tight loop,this is doable would inefficient

The forth command record the current time in the "start" structure
The fifth command write the file to disk
The sixth command record the current time in the "end" structure

The rest of the code calculate the time difference btw the two times,do simple math and report the time.